Head-Mounted Display Apparatus for Retaining a Portable Electronic Device with Display

ABSTRACT

Head-mounted display systems and methods of operation that allow users to couple and decouple a portable electronic device such as a handheld portable electronic device with a separate head-mounted device (e.g., temporarily integrates the separate devices into a single unit) are disclosed. The portable electronic may be physically coupled to the head-mounted device such that the portable electronic device can be worn on the user&#39;s head. The portable electronic device may be operatively coupled to the head-mounted device such that the portable electronic device and head mounted device can communicate and operate with one another. Each device may be allowed to extend its features and/or services to the other device for the purpose of enhancing, increasing and/or eliminating redundant functions between the head-mounted device and the portable electronic device.

This application is a continuation of patent application Ser. No.15/684,754, filed Aug. 23, 2017, which is a continuation of patentapplication Ser. No. 15/451,230, filed Mar. 6, 2017, which is acontinuation of patent application Ser. No. 15/274,901, filed Sep. 23,2016, which is a continuation of patent application Ser. No. 15/246,340,filed Aug. 24, 2016, which is a continuation of patent application Ser.No. 15/215,122, filed Jul. 20, 2016, which is a continuation of patentapplication Ser. No. 14/862,036, filed Sep. 22, 2015, which is adivision of patent application Ser. No. 14/604,059, filed Jan. 23, 2015,which is a continuation of patent application Ser. No. 12/242,911, filedSep. 30, 2008, which are hereby incorporated by reference herein intheir entireties. This application claims the benefit of and claimspriority to patent application Ser. No. 15/684,754, filed Aug. 23, 2017,patent application Ser. No. 15/451,230, filed Mar. 6, 2017, patentapplication Ser. No. 15/274,901, filed Sep. 23, 2016, patent applicationSer. No. 15/246,340, filed Aug. 24, 2016, patent application Ser. No.15/215,122, filed Jul. 20, 2016, patent application Ser. No. 14/862,036,filed Sep. 22, 2015, patent application Ser. No. 14/604,059, filed Jan.23, 2015, and patent application Ser. No. 12/242,911, filed Sep. 30,2008.

BACKGROUND OF THE INVENTION

This invention is directed to a head-mounted device constructed toreceive a portable electronic device having a screen.

Using head-mounted devices, a user may view media provided by a portableelectronic device. For example, a user may couple a personal electronicdevice, such as the iPod™ available from Apple Inc. of Cupertino,Calif., or the iPhone™ also available from Apple Inc. of Cupertino,Calif., to the head-mounted device via a cable or wire. Such aconfiguration can allow the user to view media on a private display,while the media is provided by a personal handheld device.

Sometimes, however, a wired connection may be inconvenient andcumbersome for the user in certain situations (e.g., the user mustseparately hold multiple devices and deal with cables).

In addition to being unwieldy, the coupled system often utilizesredundant features, which are not necessary when using the devicestogether. By way of example, each device utilizes a display screen,which adds cost, size, weight, and complexity to the entire system.

Accordingly, there is a need for an improved head-mounted displaysystem, particularly a system that temporarily integrates or merges bothmechanically and electronically a head-mounted device with a portableelectronic device.

SUMMARY OF THE INVENTION

In accordance with the invention, a head-mounted display system andmethod of operation are provided in which the system can allow users tocouple and decouple a portable electronic device with a head-mounteddevice. The portable electronic device may for example be physicallyand/or operatively coupled and decoupled with the head-mounted device.In some embodiments, the two devices may be considered temporarilyintegrated.

In accordance with one embodiment of the invention, there is provided ahead-mounted device that is worn on a user's head. The head-mounteddevice may include a frame that is configured to physically receive andcarry a portable electronic device. The frame may place a display screenof the portable electronic device in front of the user's eyes. Thedisplay screen of the portable electronic device may act as the primarydisplay screen of the head-mounted device such that the display screenof the portable electronic device is primarily used to view image basedcontent when the head-mounted display device is worn on the user's head.

In accordance with another embodiment of the invention, there isprovided a method of controlling a portable electronic device having ascreen capable of presenting image based content, wherein the portableelectronic device presents the image based content in a normal viewingmode that substantially fills the screen during normal use of theportable electronic device. The method may include detecting aconnection with a head-mounted device that is capable of being worn on auser's head, wherein the connection indicates that the portableelectronic device is operatively coupled and physically carried by thehead-mounted device. The method may also include switching to ahead-mounted mode on the screen of the portable electronic device.

In accordance with yet another embodiment of the invention, there isprovided a method for displaying image based content on a head-mounteddevice. The method may include coupling a portable electronic device tothe head-mounted device such that a screen of the portable electronicdevice faces a user. The method may also include providing aninstruction to play back image based content stored on the portableelectronic device, wherein the image based content is displayed on thescreen of the portable electronic device. The method may also includeadjusting the image based content displayed on the screen for close upviewing.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features of the present invention, its nature andvarious advantages will be more apparent upon consideration of thefollowing detailed description, taken in conjunction with theaccompanying drawings in which:

FIG. 1 shows a simplified diagram of a head-mounted display system inaccordance with embodiments of the invention;

FIGS. 2A and 2B show simplified schematic diagrams of a head-mounteddisplay system in accordance with embodiments of the invention;

FIGS. 3A-3C show perspective views of a head-mounted device inaccordance with embodiments of the invention;

FIG. 4 shows an alternative configuration of a head-mounted device inaccordance with embodiments of the invention;

FIG. 5 shows a configuration for sliding a portable electronic deviceinto a head-mounted device in accordance with embodiments of theinvention;

FIGS. 6A and 6B show top views of two exemplary head-mounted displaysystems in accordance with embodiments of the invention;

FIG. 7 shows a side view of a head-mounted device coupled to a portableelectronic device in accordance with embodiments of the invention;

FIGS. 8A-8D show illustrative features for exemplary head-mounteddevices in accordance with embodiments of the invention;

FIGS. 9A-9C show a head-mounted device in accordance with embodiments ofthe invention;

FIGS. 10A and 10B shows an exemplary portable electronic device inaccordance with embodiments of the invention;

FIGS. 11A-11F show several examples of creating image frames within adisplay screen in a head-mounted display system in accordance withembodiments of the invention;

FIGS. 12A-12C show some exemplary display configurations for ahead-mounted display system when displaying external and stored imagebased content in accordance with embodiments of the invention;

FIG. 13 shows a simplified block diagram of an illustrative head-mounteddisplay system in accordance with embodiments of the invention;

FIG. 14 shows a flowchart of an illustrative process for displayingimage based content on a portable electronic device in accordance withembodiments of the invention;

FIG. 15 shows a flowchart of an illustrative process for adjusting imagebased content displayed on a portable electronic device in accordancewith embodiments of the invention;

FIG. 16 shows a flowchart of an illustrative process for displayingimage based content on a head-mounted device in accordance withembodiments of the invention;

FIGS. 17A and 17B show flowcharts of an illustrative process fordisplaying external image based content in accordance with embodimentsof the invention;

FIG. 18 shows a process for displaying images in a head-mounted displayapparatus to a user in accordance with embodiments of the invention;

FIG. 19 shows a process for displaying images in a head-mounted displayapparatus to a user in accordance with embodiments of the invention;

FIG. 20 shows a process for displaying images on a screen of a portableelectronic device in accordance with embodiments of the invention; and

FIG. 21 shows a process for displaying one or more image frames on ascreen of a portable electronic device coupled to a head-mounted displayapparatus in accordance with embodiments of the invention.

DETAILED DESCRIPTION OF THE INVENTION

This invention generally pertains to head-mounted display systems andmethods of operation that allow users to couple and decouple a portableelectronic device such as a handheld portable electronic device with aseparate head-mounted device (e.g., temporarily integrates the separatedevices into a single unit). The head-mountable device may be configuredto be worn on a user's head. The portable electronic device, on theother hand, may be configured to provide specific features/applicationsfor use by a user. Examples of portable electronic devices include mediadevices, personal digital assistants (“PDAs”), telephones, and the like.

One aspect of the invention relates to physically coupling (e.g.,mechanically) the portable electronic device to the head-mounted devicesuch that the portable electronic device can be worn on the user's head.In some embodiments, the head-mounted device may include a supportstructure that supports, secures, and carries the portable electronicdevice (e.g., physically integrated as a single unit). The supportstructure may also help place a display of the portable electronicdevice relative to a user's eyes when the integrated system is worn onthe user's head. In one example, the support structure helps define adocking area for receiving and retaining the portable electronic device.

Another aspect of the invention relates to operatively coupling (e.g.,electronically) the portable electronic device to the head-mounteddevice such that the portable electronic device and head mounted devicecan communicate and operate with one another. The head mounted devicemay include, for example, interface mechanisms that enable communicationand operatibility between the portable electronic device and thehead-mounted device. The interface mechanisms may, for example, includeelectrical mechanisms such as connectors or chips that provide wired orwireless communications. In some embodiments, the head-mounted devicemay include a connector that receives a corresponding connector of theportable electronic device. The connector may, for example, be locatedwithin a docking area of the head mounted device such that the portableelectronic device operatively connects when the portable electronicdevice is placed within the docking area. The interface mechanisms mayalso include optical interface mechanisms, such as lenses, etc., thatprovide optical communications for proper viewing of a display of theportable electronic device.

Another aspect of the invention relates to allowing each device toextend its features and/or services to the other device for the purposeof enhancing, increasing and/or eliminating redundant functions betweenthe head-mounted device and the portable electronic device physicallyand/or operatively coupled thereto. In some embodiments, the headmounted device utilizes components of the portable electronic devicewhile in other embodiments, the portable electronic device utilizescomponents of the head mounted device. In one example, the head-mounteddevice utilizes the display of the portable electronic device as theprimary display for displaying media (e.g., image based content). Thatis, the head-mounted device does not include a main viewing displayscreen and instead utilizes the screen of the portable electronic deviceto act as the main or primary display when the portable electronicdevice is coupled thereto.

Embodiments of the invention are discussed below with reference to FIGS.1-21. However, those skilled in the art will readily appreciate that thedetailed description given herein with respect to these figures is forexplanatory purposes, as the invention extends beyond these limitedembodiments.

FIG. 1 shows a simplified diagram of a head-mounted display system 100,in accordance with one embodiment of the present invention. Thehead-mounted display system 100 can include head-mounted device 102 andportable electronic device 104 that are separate devices which can betemporarily coupled together to form an integrated unit, which can beworn on a user's head.

Portable electronic device 104 may be widely varied. For example,portable electronic device 104 may be configured to provide specificfeatures and/or applications for use by a user. Portable electronicdevice 104 may be a light weight and small form factor device so that itcan easily be supported on a user's head. In most embodiments, theportable electronic device includes a display 106 for viewing imagebased content.

In one embodiment of the present invention, portable electronic device104 may be a handheld electronic device such as a portable media player,cellular telephone, internet-capable device, a personal digitalassistant (“PDA”), any other portable electronic device, or anycombination thereof. In another embodiment of the present invention,portable electronic device 104 can be a device that has the combinedfunctionalities of a portable media player and a cellular telephone,such as an iPhone™ sold by Apple Inc. of Cupertino, Calif. In yetanother embodiment of the present invention, the portable electronicdevice 104 can be a dedicated media player such as an iPod™ sold byApple Inc. of Cupertino, Calif. Examples of iPods™ may include Touch™,Nano™, Classic™, and the like.

Head-mounted device 102 may be configured to be worn on a user's head.The head-mounted device 102 may be configured to receive and carryportable electronic device 104 in order to create head-mounted displaysystem 100. In some embodiments, head-mounted device 102 may include aframe or carrier 108 that supports and holds the portable electronicdevice 104 thereby allowing portable electronic device 104 to be worn ona user's head (e.g., glasses/goggles form factor). As should beappreciated, because portable electronic device 104 can be carried byhead-mounted device 102, the user can relax while viewing image basedcontent on the head-mounted device because he does not have to hold ontothe portable electronic device.

The frame or carrier 108 may for example be configured to be situated infront of a user's face. As a result, screen 106 of the portableelectronic device 104 may be oriented towards the user's eyes whenhead-mounted display system 100 (the head-mounted device 102 includingthe portable electronic device 104) is worn on the user's head. In oneembodiment, the head-mounted device 102 may not include a main viewingdisplay but may rather rely on screen 106 as the primary display and/orthe only display of system 100. A primary display may be the primarycomponent used for displaying image based content (e.g., media such asmovies, TV shows, and graphical user interfaces associated with varioussoftware applications) when the system is worn on the user's head. Byusing such a configuration, head-mounted display system 100 may reduceredundancies, which can add cost, weight and complexity to the system.

The frame or carrier 108 may define or include a docking area 110 forreceiving and retaining the portable electronic device 104. The dockingarea 110 may be widely varied. The docking area 110 define an area intowhich a portion or the entire portable electronic device 104 may beplaced. The docking area 110 may also include one or more retentionfeatures for holding and securing the portable electronic device withinthe docking area. The docking area may be defined by walls that surroundsome portion of the portable electronic device (e.g., exteriorsurfaces). The retention features may for example include rails,channels, snaps, detents, latches, catches, magnets, friction couplings,doors, locks, flexures, and the like.

In one embodiment, the system 100 may include a detection mechanism 116a and/or 116 b that provides inputs to the portable electronic devicewhen it is carried by the head mounted device so that the portableelectronic device is configured for head-mounted use. By way of example,the detection mechanism 116 may be an electrical connection, a sensorsuch as a proximity sensor or infrared (“IR”) detector and/or the like.In one example, once the portable electronic device is placed relativeto the head mounted device, the portable electronic device goes into ahead mounted mode (i.e., configures the display screen for HMD viewing).

The head-mounted display system 100 can include a communicationinterface that provides data and/or power communications between theportable electronic device 104 and the head-mounted device 102. Thecommunication interface may be wired or wireless. In some embodiments,the head mounted device 102 may include a connector 118 a that mateswith a corresponding connector 118 b of the portable electronic devicewhen the portable electronic device is placed within the docking area110.

The protocol under which the devices 102 and 104 communicate via thecommunication interface may be widely varied. By way of example, thecommunication protocol may be a master/slave communication protocol,server/client communication protocol, peer/peer communication protocol,and the like. Using a master/slave communication protocol, one of thedevices is a master and the other is a slave. The master controls theslave. Using a client/server communication protocol, a server programresponds to requests from a client program. The server program mayoperate on the portable electronic device or the head mounted device.Using a peer to peer communication protocol either of the two devicescan initiate a communication session.

Generally, the communication session begins when the two devices arecoupled together and powered up. In some cases, the devices may goimmediately into a HMD mode when connected. For example, based on adefault settings, the portable electronic device may be configured forclose up head mounted viewing (either directly or via instructions fromthe head mounted device 102). Further, input devices, output devices,sensors, and other electrical systems on both devices may be activatedor deactivated based on the default settings. Alternatively, the usermay be prompted with a control menu for setting up the system when theyare operatively coupled together via the communication interface 107.

Head-mounted display system 100 can include a variety of features, whichcan be provided by one or both devices of the system when they areconnected and in communications with one another. For example, eachdevice may include one or more of the following components: processors,display screen, controls (e.g., buttons, switches, touch pads, and/orscreens), camera, receiver, antenna, microphone, speaker, batteries,optical subassembly, sensors, memory, communication systems,input/output (“I/O”) systems, connectivity systems, cooling systems,connectors, and/or the like. If activated, these components may beconfigured to work together or separately depending on the needs of thesystem. In some cases, features may be turned off entirely if not neededby the system.

In some embodiments, the head-mounted device 102 may provide additionalfeatures for the head-mounted display system 100. In one example, thehead-mounted device 102 can provide additional functionality to theportable electronic device 104. For instance, the head-mounted device102 may include buttons, switches, dials, touch devices, or any othersuitable controls. These controls may match or mimic the controls foundon the portable electronic device. In addition, the head-mounted device102 can include a battery to extend the life of the portable electronicdevice 104. Furthermore, the head-mounted device 102 can include acooling system for cooling down the portable electronic device 104.Moreover, the head-mounted-device 102 can include memory, for example,to store image based content, which can be displayed on the screen 106of the portable electronic device 104. Persons skilled in the art willappreciate that any other suitable functionality may be extendedincluding additional circuitry, processors, input/output, optics, and/orthe like.

In some embodiments, the system including one or both of thehead-mounted device 102 and the portable electronic device can includesensors to detect various signals. For example, accelerometers on thehead-mounted device and/or the portable electronic device can be used todetect the user's head movements. In this example, the head-mounteddevice 102 can associate a particular head movement with a command forcontrolling an operation of the head-mounted display system 100. As yetanother example, the head-mounted system 100 can utilize a proximitysensor on one or both of the head mounted device and portable electronicdevice to detect and identify the relationship between the two devicesor to detect and identify things in the outside environment. As yetanother example, the head-mounted system 100 can utilize a microphone onone or both of the head mounted device and portable electronic device todetect and identify voice commands that can be used to control theportable electronic device 104. As yet another example, the head-mountedsystem 100 can utilize a camera on one or both of the head mounteddevice and portable electronic device to capture images and/or video.The image based content may for example be viewed on the display of thehead mounted display system. In one embodiment, the image based contentmay be viewed in addition or alternatively to image based media contentplaying on the display. In one example, the captured content may beviewed in a picture in picture window along with the media basedcontent.

While the two devices 102 and 104 may provide redundant features to thesystem 100, in one embodiment, at least some of the redundancies arereduced or eliminated. For example, if the portable electronic deviceincludes a camera then the head mounted device may not. Additionally oralternatively, if the portable electronic device does not include amicrophone then the head mounted device may include one. As should beappreciated, this may reduce cost and complexity of the system 100. Itmay also keep the form factor of both devices 102 and 104 small andlightweight. In addition, it can extend the functionalities of eachdevice.

In some embodiments, the head-mounted device 102 may not include aprimary display but rather rely on a screen 106 of the portableelectronic device 104 as the primary display and/or the only display ofthe system 100. A primary display may, for example, be the primarycomponent used for viewing image based content when the system is wornon the user's head. Image based content may for example include but notlimited to media such as movies, TV shows, and graphical user interfacesassociated with various software applications.

In accordance with one embodiment, when the portable electronic device104 is used in the head mounted device 102, the size and possibly theresolution of the image based content being displayed on the screen 106may be adjusted for close up viewing (e.g., via the detection mechanismor the connection interface). For example, the image based content maybe displayed in a viewing region that is configured smaller than theactual screen size (e.g., due to how close it is placed to the user'seyes) and/or the resolution may be increased/decreased relative tonormal portable electronic device viewing to provide the best close upviewing experience. In one implementation, the viewing region isconfigured to fill the entire field of view of the user. In anotherimplementation, the viewing region is configured to be less than thefield of view of the user.

In one embodiment, the displayed content may be split into multipleimages frames. For example, the displayed content may be split into twoimage frames (e.g., a left and right image frame for the left and righteye of the user). With two image frames, the system can performstereoscopic imaging. Stereoscopic imaging attempts to create depth tothe images by simulating the angular difference between the imagesviewed by each eye when looking at an object, due to the differentpositions of the eyes. This angular difference is one of the keyparameters the human brain uses in processing images to create depthperception or distance in human vision. In one example, a single sourceimage, which can be provided by the HMD, PED or external medium, isprocessed (e.g., either by the HMD or PED) to generate left image dataand right image data for viewing. This can be accomplished by theportable electronic device and/or the head mounted device depending onthe needs of the system.

The image frame(s) can also have different aspect ratios depending onthe primary application of the displayed content. For example, an aspectratio of 4×3 (i.e., similar to a television screen) may be used fortelevision programs, whereas an aspect ratio of 16×9 (i.e., similar to amovie theatre screen) may be used for watching feature films.

FIGS. 2A and 2B show simplified schematic diagrams of a head-mounteddisplay system 200 in accordance with one embodiment of the presentinvention. Head-mounted display system 200 may generally correspond tothe head-mounted display system described in FIG. 1. Head-mounteddisplay system 200 includes head-mounted device 202 that receives aportable electronic device 204 having a display screen 206. That is,portable electronic device 204 may be coupled to head-mounted device 202(as shown in FIG. 2B) and removed from head-mounted device 202 (as shownin FIG. 2A). When coupled, portable electronic device 204 is carried byhead-mounted device 202, and as a result may be worn on the user's head.

Portable electronic device 204 can be, for example, a portable mediaplayer, cellular telephone, internet-capable device, personal organizer,any other portable electronic device, or any combination thereof. In oneembodiment of the present invention, the portable electronic device 204can be a device that has the combined functionalities of a portablemedia player and a cellular telephone, such as an iPhone™ sold by AppleInc. In another embodiment of the present invention, the portableelectronic device can be a dedicated media player such as an iPod™ soldby Apple Inc. Examples of iPods™ include for example the Touch™, Nano™,Classic™, and the like.

The display screen 206 of the portable electronic device 204 istypically oriented towards the user's eyes when the head mountabledevice 202 including the portable handheld device 204 is worn on theuser's head (towards the eyes of the user). As such, the display screen206 may be used to view content when being worn. In one particularembodiment, the head mountable device 202 does not include its own maindisplay and rather relies on the display 206 of the portable electronicdevice 204 coupled thereto to provide image based content (therebyreducing redundancies, which can add cost and complexity to the system).

The head mountable device 202 typically includes a frame member 208. Theframe member 208 may be supported on a user's head in a variety of waysincluding for example, ear support bars as in glasses, bands as ingoggles, helmets, straps, hats and the like. The frame member 208 mayhave a docking area or mounting region 210 for placement of the portableelectronic device 204 relative to the head mountable device 202. Theframe member 208 may be configured to support the portable electronicdevice 202 when it is mounted in the mounting region 210.

The frame member 208 is typically worn in front of the users face. Themounting region 210 is therefore configured to place the display screen206 in the correct position for viewing relative to a user's eyes (e.g.,in front of the users eyes as well as some of the distance from theuser's eyes).

The mounting region 210 may be widely varied. In one embodiment, themounting region 210 includes a cavity 212. The cavity may for examplehave walls shaped similarly to the outer shape of the portableelectronic device 204. In some cases, the cavity 212 may only includewalls that surround multiple but not all sides of the portableelectronic device 204 (e.g., at least two sides, three sides, fourssides, five sides,). The cavity 212 may form a void if five sides areused. In other cases, the cavity 212 may include walls on all sides(e.g., six sides). In any of these implementations, the walls mayinclude open areas depending on the needs of the system. Alternatively,the cavity 212 may be formed with corners that match the corners of theportable electronic device 204.

The manner in which the portable electronic device 204 is placed withinthe cavity 212 may be widely varied. In one implementation, the portableelectronic device 204 may be rotated into the cavity (e.g., by insertinga first end into the cavity 212 and thereafter rotating about thecontact point). In another implementation, the portable electronicdevice 204 may be press fit into the cavity 212 (e.g., by pushing theportable electronic device into the shaped cavity). In yet anotherimplementation, the portable electronic device 204 may be slid into thecavity (e.g., through a slot in one of its sides).

The head mountable device 202 can include one or more retaining features214 for helping secure the portable electronic device 204 within themounting region 210 of the head mountable device 202. The retainingfeatures 214 may for example include doors, tabs, slots, detents, lips,snaps, flexures, friction couplings, clips, latches, catches, and/or thelike.

The frame may include an opening 215 which provides access for viewingthe display screen 206 when the portable electronic device is placedwithin the cavity 212. The opening 215 may be widely varied. In oneembodiment it coincides with the shape/dimensions of the display screen206. In another embodiment, it may be smaller than the shape/dimensionsof the display screen 206. In either embodiment, it may include a bezel216 that covers or overlays select portions of the portable electronicdevice 204 when the portable electronic device 204 is mounted within thehead mountable device 202. That is, the bezel 216, which may be aseparate component or a portion of one of the walls, can form or definethe viewing region (blocking some of the device housing and/or thedisplay screen). The bezel 216 may for example extend around all or someof the edge of the opening 215. In some cases, the opening 215 mayinclude a transparent window disposed therein.

Shielding portions of the display screen 206 may be advantageous forenhancing the viewing experience and possibly reducing the energy burdenof the portable electronic device (e.g., the shielded display pixels canbe turned off). By way of example, the shielding portions may be black.The shielding portion may in some cases form a retaining structure. Forexample, it may be a fixed wall that forms a lip that covers a frontportion of the portable electronic device when it is mounted or it maybe a door that swings from an open position allowing the portableelectronic device 204 to be inserted or removed and a closed positionthat locks the portable electronic device 204 in place.

The display screen 206 of the portable electronic device 204 may bewidely varied and as such so may the head mountable device 202configured for receiving the portable electronic device 204 (asmentioned above). In one implementation, the display screen 206 may forexample be a full screen display that extends from edge to the edge ofthe portable electronic device 204. In another implementation, thedisplay screen 206 may for example be a substantially full screendisplay such that a majority of the display screen 206 fills the frontsurface of the portable electronic device 204. In anotherimplementation, the display screen 206 may for example be a partialscreen display. For example, the partial screen display may only beoriented in an upper half of the portable electronic device 204.Further, the display screen 206 may be symmetrically centered on thefront face of the portable electronic device 204 or alternatively it maybe offset towards one side. In all of these implementations, the entirescreen or portions of the screen can be covered to provide the desiredscreen size/shape for the viewing experience.

When coupled, the distance of the display screen 206 relative to theuser's eyes may be widely varied. In small form factor head mountabledevices (e.g., low profile), the display screen 206 of the portableelectronic device 204 may be placed fairly close to the user's eyes. Theplacement of the screen 206 may be controlled by the surfaces ofmounting region 210 and more particularly the walls of the cavity 212.

Because the screen 206 may be so large compared to the viewing distance,the system 200 may be configured to modify the image based contentdisplayed on the display screen 206 so that is can be easily viewed. Forexample, the system 200 may adjust the size of the displayed content tobe less than the size of the display screen (see, e.g., FIG. 11A). Thesystem 200 may also adjust the resolution of the viewable content. Forexample, the resolution may be increased/decreased to compensate for howclose the eyes are to the display screen 206.

It should also be emphasized that the system 200 is not limited todisplaying one image frame. In some cases, it may be beneficial toprovide multiple redundant image frames (dual image frames) in order toimplement stereo like viewing (see for example FIGS. 11B-11F). In casessuch as these, the image frames may be displayed at two locations withinthe display screen 206. The timing or image characteristics of the dualimage frames relative to one another may be varied to provide anenhanced viewing effect.

While in most cases, the single image frame or dual image frame aretypically centered within the screen 206, in some cases, it may bedesirable to adjust the position of the displayed content within thedisplay screen 206 in order to better align the image/content to theuser (e.g., align or center the displayed image relative to where theeyes should be or to where the eyes actually are). The alignment may bebased on a particular location relative to the head mountable device 202and/or based on the position of the user's eyes relative to the screen206. For example, the image frames(s) may be centered relative to a nosesupport of the frame member 208, and/or the head mountable device 202may include a means for discerning the location of the eyes (e.g.,optical sensor). As shown by arrows in FIG. 11C, the position of thedual images may be adjusted together. Furthermore, as shown by thearrows in FIG. 11D, the position of the dual images may be adjustedrelative to one another. For example, one may be adjusted upward whilethe other is adjusted to the right. Furthermore, although not shown, theorientation of each image frame may also be adjusted together orseparately (rotation).

Although not shown, the system 200 may include an optical subassemblyfor helping properly display the one or more image frames to the user.That is, the optical subassembly may help transform the image frame(s)into an image(s) that can be viewed by the human eye. Opticalsubassembly may for example focus the images from the respective imageframe(s) onto the user's eyes at a comfortable viewing distance.

The optical subassembly may be disposed between the display screen 206and the user's eyes. The optical subassembly may be positioned in frontof, behind or within the opening 215 that provides viewing access to thedisplay screen 206. The optical subassembly may be supported by theframe member 208. For example, it may be attached to the frame membervia any suitable means including for example screws, adhesives, clips,snaps, and the like.

The optical subassembly may be widely varied. The optical subassemblymay include various optical components that may be static or dynamiccomponents depending on the needs of the system. The optical componentsmay include, for example, but not limited to lenses, light guides, lightsources, mirrors, diffusers, and the like. The optical sub assembly maybe a singular mechanism or it may include dual features, one for eacheye/image area. In one implementation, the optical sub assembly can beformed as a panel that overlays the access opening 215. The panel may becurvilinear and/or rectilinear. For example, it may be a thin flat panelthat can be easily carried by the frame member 208 and easily supportedon a user's head. If dynamic, the optical subassembly 219 may bemanually or automatically controlled.

Specific examples of optical sub assemblies may include aspherical anddiffractive optical arrangements. Aspherical optical arrangements mayinclude various types of prism and/or lens arrangements (e.g., coaxialarrangement of lenses, a coaxial arrangement of a set of lenses arrangedalong a common axis, a concave mirror arrangement, a free shaped prismarrangement). Diffractive optical arrangements may include various typesof arrangements that bend and spread light (e.g., a light-guided opticalelement, or a binocular light-guided optical element, both of which canbe made of planar transparent glass or plastic).

In some embodiments, the optical subassembly may include one or moreoptical modules that may be operative to adjust or modify the displayedmedia based on any suitable criteria (e.g., as standalone modules orusing circuit board). For example, optical modules may be operative toresize or otherwise modify an image such that the image appears to bedisplayed at a greater distance than the distance between opticalmodules and the user's eyes (e.g., the image appears to be viewed from 4meters and not 20 mm). As another example, optical modules may beoperative to generate media displays that correspond to standarddefinition or high definition images. In some embodiments, left andright optical modules may generate and display different imagesassociated with the received media. For example, optical modules (e.g.,upon receiving appropriate instructions from the HMD or PED), may offsetleft and right images so that the user is given the illusion of viewingmedia in three dimensions. The optical modules may identify theappropriate left and right images using any suitable approach, includingfor example receiving separate left and right images from the mediasource, or generating left and right images from the single mediaprovided by the media source. As another example, left and right opticalmodules may adjust the images displayed to account for limitations orcorrections (e.g., by glasses or contact lens) of the user's eyesight.For example, if a user is myopic in one eye, the optical moduleassociated with that eye may modify the image displayed to correct theuser's myopia. The system may determine the correction required for eacheye, if any, using any suitable approach. For example, the user mayenter a glasses or contact lens prescription that indicates the requiredcorrection. As another example, each optical module may automaticallyanalyze the user's eyes, and determine the adjustment needed based onthe analysis. This may allow users who normally wear glasses to usesystem without their glasses, which may lead to discomfort.

The system 200 may include a detection mechanism for alerting theportable electronic device 204 that it has been mounted or is otherwisecarried by a head mountable device 202. Once detected, the portableelectronic device 204 can be placed in a head mounted display (HMD)mode. The HMD mode may be based as a default and/or on user preferences.The HMD mode can be set up to configure the portable electronic device204 for HMD viewing. For example, the HMD mode may adjust theimage/content so that it can be viewed up close (as mentioned above). Ifuser preferences are used, the user may be able to make adjustments asneeded. Since adjustments may be difficult for the user, in some cases,the HMD and/or portable electronic device may include mechanisms forautomatically configuring the image location and size. For example,either device may include sensors for detecting the distance to the eyesand the position of the eyes. As should be appreciated, each user's eyesare oriented differently. For example some eyes are located closetogether while others are more spread out. The optimal viewing positionsof the displayed images can be determined and then the viewing positionscan be adjusted. The same can be done for resolution. Although, allowingthe user to adjust resolution may be beneficial as this is a moredifficult measurement to make since eyes can focus differently. By wayof example, the portable electronic device 204 and/or the HMD 202 mayinclude cameras that can reference where the eyes are located relativeto the HMD 202. Alternatively, the portable electronic device may becomea slave to the head mountable device such that the head mountable devicecontrols the operation of the portable electronic device once it isinserted into the head mountable device.

The system 200 can include a communication interface that provides dataand/or power communications between the portable electronic device 204and the HMD 202. The communication interface may be wired or wireless.

If wired, the HMD 202 may include a connector 228 that receives acorresponding connector 229 of the portable electronic device 204 whenthe portable electronic device 204 is supported/carried by the headmountable device 202. In most cases, the connectors mate when the deviceis placed within the mounting region 210, and more particularly whenplaced within the cavity 212. By way of example, the connectors may mateas the portable electronic device is rotated, slid, or pressed into theHMD 202. The connectors may be male/female. For example, the portableelectronic device may include a female connector while the HMD 202 mayinclude a male connector. In this particular case, the male connector isinserted into the female connector when the devices are coupledtogether. The connectors may be widely varied. The connectors may be lowprofile connectors. The connectors may for example correspond to USBand/or FireWire. The connector may also be a 30 pin connector used iniPods™ and iPhones™ manufactured by Apple Inc. of Cupertino, Calif. Insome cases, the cavity/connector combination may generally define adocking station for the portable electronic device.

If wireless, the devices 202 and 204 may each include a wireless chipset that transmits and/or receives (transceiver) the desired signalsbetween the devices 202 and 204. Examples of wireless signal protocolsinclude Bluetooth™, 802.11, RF, and the like. Wireless connections mayrequire that wireless capabilities be activated for both thehead-mounted device and the portable electronic device. However, such aconfiguration may not be possible or may be intermittent when thedevices are being used in certain locations as for example on anairplane.

As a result of operatively coupling the device 204 through wired and/orwireless means, features of the HMD 202 may be extended to the portableelectronic device 204 and features of the portable electronic device 204may be extended to the head mountable device 202. Furthermore, redundantfeatures may be removed from the system 200. For example, the HMD 202may be configured without a display and instead utilize the display ofthe portable electronic device 204.

The number of additional features 230 provided by the HMD 202 may bewidely varied. The features may be disposed internally or externally. Inone example, the head-mounted device 202 can provide additionalfunctionality to the portable electronic device 204. For example, theHMD 202 may include additional controls in the form of buttons,switches, dials, touch devices, etc. In addition, the head-mounteddevice 202 can include a battery to extend the life of the portableelectronic device 204. Furthermore, the head-mounted device 202 caninclude a cooling system for cooling down the portable electronic device204. Moreover, the head mounted device 202 can include memory as forexample to store image based content, which can be displayed on thescreen 206 of the portable electronic device 204. The head-mounteddevice 202 can also include sensors to detect various signals.

The head-mounted display system 200 can include several modes ofoperation. For example, the head-mounted display system 200 can operatein an internal viewing mode, which allows the user to watch image basedcontent playing on the portable electronic device. As another example,the head-mounted display system can operate in a picture in picture(“PIP”) mode. In the PIP mode, a PIP image frame can be partiallyoverlaid on top of the standard viewing image frame(s).

As yet another example, the head-mounted display system can operate inan external viewing mode, which allows the user to view the outsideworld. The head-mounted display system 200 can provide external viewingof the user's surroundings using any suitable approach. For exampleusing a camera of the portable electronic device 204 or using a separatecamera disposed on the outside of the head mountable device 202. In theillustrated embodiment, the portable electronic device includes a cameraon its backside opposite the display screen and the head mounted deviceincludes an aperture within the mounting region that provides cameraaccess to outside the head mounted device. In this manner, the cameramay capture image based data that is in front of the user and furtherimage frames corresponding to image based content received from thecamera can be displayed in the standard image frame(s).

In some embodiments, the head-mounted display system 200 can detect whenthe user has selected an option to view external image based content(e.g., the user may have selected a control, issued a voice command, ormade a head movement). In some embodiments, the head-mounted displaysystem 200 can provide external viewing based on an indication receivedfrom the user's surroundings. For example, sensors on the head-mounteddisplay system can detect objects or sound around the user and enableexternal viewing.

The head-mounted display system 200 can make adjustments in response todetecting objects or sound. For example, arrows can be displayed on thedisplay screen of the head-mounted display that corresponds to thedirection of detected sound. As another example, the head-mounteddisplay system 200 can automatically pause or stop the stored imagebased content that is currently playing. As yet another example, thehead-mounted display system 200 can lower the volume of the stored imagebased content so that the user can hear the external sounds. In somecases, along with lowering the volume of the stored image based content,the head-mounted display system 200 can provide close captioning of thestored image based content.

In some embodiments, the display screen 206 can be utilized to extendthe view of the displayed content. For example, because the screen 206is larger than the reduced sized image frame(s), portions of the screen206 surrounding the image frame(s) may be used to extend the feel of thedisplayed images to produce enhanced viewing effects. The extendedportions may be colors associated with the edge of the displayed image.The secondary effect may be continuous or segmented. It may extend fromall edges or just corners. Its characteristics including speed can alsobe adjusted to produce distinct effects. Alternatively or additionally,the images may be made larger and smaller to produce a viewing effect.

Other examples include shaking, shifting, etc. See, for example, FIG.11F, which shows a display screen broken up into a set of dual primaryimage regions (showing the desired content) and a set of secondaryvisual effect regions, which surround at least a portion of each of theprimary image regions, and which extend and provide enhancements to theimages displayed in the primary image regions.

The head-mounted device 204 can also serve a secondary purpose for theportable electronic device. For example, the head-mounted device 204 canserve as a case for the portable electronic device 202. As anotherexample, the head-mounted device 204 can serve as a display stand forthe portable electronic device 202.

These and other embodiments will be described in greater detail below.

FIGS. 3A-3C show various views of a head-mounted device in accordancewith one embodiment of the invention. The head-mounted device 300 maygenerally correspond to the head-mounted device 102 of FIG. 1 orhead-mounted device 204 of FIG. 2. Head-mounted device 300 can includeany suitable component.

The head mounted device 300 may include a face plate 302. Face plate 302may be typically worn in front of the user's face. Faceplate 302 may besupported on a user's head using a variety of components including, forexample, eyeglass temples 312. Alternatively, goggle bands, helmets,straps, hats and the like may be used.

Faceplate 302 can form the outer surface of head-mounted device 300.Faceplate 302 can be constructed into any suitable shape and dimension.In the example shown in FIGS. 3A and 3B, faceplate 302 generally has anouter shape configured to cover a user's eyes when worn. It may alsoinclude a nose recess such that it can be supported on a user's nose.This may serve as a reference surface for placing head-mounted device300 relative to a user's eyes.

Faceplate 302 can be formed from any suitable material or materials. Insome embodiments, the faceplate can be formed from lightweight materialsthat afford user comfort (e.g., plastic) while maintaining strength tosupport a portable electronic device. In some embodiments, the faceplatecan be formed from a material capable of withstanding impacts or shocksto protect the components of head-mounted device 300.

Examples of materials include composite material, glass, plastic (ABS,polycarbonate), ceramic, metal (e.g., polished aluminum), metal alloys(e.g., steel, stainless steel, titanium, or magnesium-based alloys), orany other suitable material. In some embodiments, the outer surface offrame 302 can be treated to provide an aesthetically pleasing finish(e.g., a reflective finish, or added logos or designs) to enhance theappearance of head-mounted device 300.

Face plate 302 may be a skeletal structure with minimal structure suchas walls thereby keeping it light weight and/or it may be configuredmore like a housing that can enclose various components. Face plate 302may include main body 303, which helps form the side surface of the headmounted device 300. Face plate 302 may also include a front panel 304and/or a back panel 305 that can be integral with or coupled to mainbody 303 to form the front and back surfaces of head-mounted device 300.Thus, main body 303, front panel 304, back panel 305 can cooperate toform the outer structure of head-mounted device 300.

Main body 303, front panel 304 and back panel 305 can be formed from anysuitable material as mentioned above. In some embodiments, the threestructures are formed from similar materials. In other embodiments, thethree structures are formed from dissimilar materials. Each has needsthat may be taken into account when designing the head mounted device.For example, the main body may be formed from a structure material witha structural configuration thereby providing central support to the headmounted device 300 while the front and back panels may be formed amaterial capable of withstanding impacts or shocks to protect thecomponents of head-mounted device 300.

Main body 303, front panel 304 and back panel 305 can be constructedinto any suitable shape. In one example, the main body takes the shapeof the eyes and nose area of the face and the front and back panels 304and 305 are substantially planar surfaces. As another example, front andback panels 304 and 305 can be curved surfaces that generally follow thecontours of a user's face.

In one embodiment, the main body, front and/or back panels 303, 304, and305 may cooperate to form an internal cavity 332 for placement of aportable electronic device relative to head-mounted device 300. Forexample, the main body, front and back panels can form side walls thatsurround the portable electronic device thereby helping to retain theportable electronic device within the head-mounted device 300. Cavity332 may be configured to place the screen of the portable electronicdevice in the correct position for viewing relative to a user's eyes(e.g., in front of the users eyes and/or some distance away from theuser's eyes). In one embodiment, main body 303 of head-mounted device300 may be configured to provide the main support of a portableelectronic device when the portable electronic device is held within thecavity 332.

Cavity 332 can be defined by walls that may have a shape similar to theouter shape of the portable electronic device. In one example, the innershape of the cavity 332 substantially conforms to the outer shape of aportable electronic device. Cavity 332 may receive the entire portableelectronic device or only a portion of the device. For example, cavity332 may include walls that surround multiple but not all sides of theportable electronic device (e.g., at least two sides, three sides, fourssides, or five sides). Thus, cavity 332 may receive only a displayportion of the portable electronic device while leaving the rest of theportable electronic device outside of the head-mounted device. In somecases, the cavity 332 may form a void if five or six sides are used. Inany of these implementations, the walls of cavity 332 may include openareas depending on the needs of the system. Persons skilled in the artwill appreciate that cavity 332 may match with the portable electronicdevice in any suitable manner.

In some embodiments, head-mounted device 300 can include an adjustablemating mechanism such that the portable electronic device can fitregardless of the size of the device or the presence or absence of acase used for the device (e.g., soft or hard case). For example, theshape and dimensions of the cavity may be physically adjusted so as tofit different portable electronic devices. Moreover, the cavity may beoversized and include a separate insert for placement therein. The innershape of the insert can be designed for a particular portable electronicdevice while the outer shape may correspond to the shape of the cavity.As another example, cavity 332 can be made from a deformable materialthat can conform to the shape of the portable electronic device (e.g.,foam). In addition, the material can also be suitable for pooling heataway from the portable electronic device.

As shown in FIGS. 3B and 3C, the back panel 305 can include an opening334 that coincides with a display screen of the portable electronicdevice (e.g., thereby allowing at least the desired viewing portion ofthe display screen to be viewed while the portable electronic device isdisposed within the cavity 330). In one embodiment, the opening 332 issized similarly to the portable electronic device. In anotherembodiment, the opening 332 is sized similarly to the display screen ofthe portable electronic device (which is typically smaller than theoverall dimensions of the portable electronic device. In anotherembodiment, the opening 332 is sized smaller than the display screen ofthe portable electronic device. For example, it may be configured to bethe size necessary for viewing smaller image frames that are displayedwithin the display screen (whether singular or double). In fact, if dualimage frames are used, then the opening may be configured as dualopenings rather than one larger opening.

Although not shown in FIGS. 3A-3C, in some cases, the HMD 300 caninclude an optical sub assembly that is disposed between the displayscreen and the user's eyes in the region of the opening 332. The opticalsub assembly may for example be disposed behind, within or in front ofthe opening 332. The optical subassembly may be supported by someportion of the main body 303 front panel 304 and/or back panel 305. Insome cases, the optical sub assembly may simply be a transparent window.In other cases, however, the optical sub assembly may also be a morecomplicated system of optical components that enhance and improved theviewing experience (i.e., help focus the user's eyes on the image framesbeing displayed on the display screen of the portable electronicdevice).

The manner in which the portable electronic device may be placed withinthe cavity 330 of the head-mounted device 300 may be widely varied. Forexample, the portable electronic device can slide into, be clipped on,be rotated into, and/or be press fit into the head-mounted device. Inthe illustrated embodiment, the face plate 302 includes a slot 340 suchthat the portable electronic device can be slid into the cavity 330. Theslot 340 may for example be positioned on the right (as shown), left,top and bottom side of the frame 302. In alternate embodiments, theportable electronic device may be rotated or press fit into the cavity330 through the opening used for viewing the display screen or if theopening is smaller than the portable electronic device, the back panelmay open like a door (e.g., a door that swings from an open positionallowing portable electronic device to be inserted or removed and aclosed position that locks portable electronic device in place).

Head-mounted device 300 may include one or more retaining features forsecuring the portable electronic device within the head-mounted device.The retaining features may, for example, include doors, tabs, slots,detents, snaps, flexures, friction couplings, latches, catches, and/orthe like. In some cases, the cavity may provide the retaining structureby being dimensioned to snuggly receive the portable electronic device(e.g., friction coupling). In some cases, the cavity may include abiasing element such as flexures or foam that squeezes the portableelectronic device when contained within the cavity. In some cases, theslot may include a door that locks the portable electronic device withinthe cavity.

Head mounted device 300 may include an internal connector 342 disposedwithin the cavity 330. The connector 342 may be configured to receive acorresponding connector of the portable electronic device when theportable electronic device is positioned within the cavity 330. Forexample, the connectors may mate as the portable electronic device isslid into the cavity 330. The walls of the cavity may properly align theconnectors during insertion. Once engaged, the head mounted device 300and the portable electronic device may be operatively coupled togethersuch that they can communicate and possibly share functionality. Theconnector 342 may be widely varied. The connector may generally providedata connections. The connector may also provide power connections. Byway of example, the connector may include for example, universal serialbus (“USB”), Firewire, or other similar connectors. In one example, theconnector is a 30 pin connector such as those used in iPods™ andiPhones™ manufactured by Apple Inc. of Cupertino, Calif.

Alternatively or additionally, the data and/or power connection can beprovided by a wireless connection. Wireless connections may be widelyvaried. For example, the devices may each include a wireless chip setthat transmits and/or receives (transceiver) the desired signals betweenthe devices. Examples of wireless signal protocols include Bluetooth™(which is a trademark owned by Bluetooth Sig, Inc.), 802.11, RF, and thelike. Wireless connections may require that wireless capabilities beactivated for both the head-mounted device and the portable electronicdevice. However, such a configuration may not be possible or may beintermittent when the devices are being used in certain locations as,for example, on an airplane.

The communication interface provided by a wired and/or a wirelessconnection can allow the user to control a portable electronic deviceusing controls provided on a head-mounted device or a remote control. Inaddition, the communication interface can provide audio out toearphones. As a result, audio from image based content playing on aportable electronic device can be directed to earphones.

Head-mounted device 300 can include earphones 316, which can provideaudio directly to the user. In some embodiments, earphones 316 can beearbuds, which can fit comfortably inside the user's ears. In theillustrated embodiment, the earphones are integrated with the eyeglasstemples 312.

In some embodiments, head-mounted device 300 can provide controls 318which can allow the user to control the portable electronic device whilewearing head-mounted device 300. Controls 318 can control any suitablefeature and/or operation of head-mounted device 300 and/or the portableelectronic device. For example, controls 318 can include navigationcontrols, display controls, volume controls, playback controls, or anyother suitable controls. Controls 318 can be located on the sidesurfaces, front surface, top surface, eyeglass temples 312, or any otheraccessible location on the periphery of head-mounted device 300.

In one embodiment, head-mounted device 300 may include one or morecontrols 318 that are functionally mapped to the buttons of the portableelectronic device when the portable electronic device is coupled.Alternatively, head-mounted device 300 may include an access opening forthe buttons of the portable electronic device.

Any suitable type of controls can be used, such as, for example, wheels,dials, buttons, switches, sliders, and touch sensors. In someembodiments, a volume control can be placed on eyeglass temples 312. Forexample, a wheel or slider can be placed proximate to one of earphones316 on eyeglass temples 312. As another example, a longitudinal touchsensor can be placed along eyeglass temples 312 (e.g., the user mayincrease the volume by sliding a finger along the touch sensor fromfront to back, and decrease the volume by sliding a finger from back tofront). As still another example, touch sensors can also be used fordisplay controls (e.g., brightness and contrast, enlarge/shrink, camerazoom, or any other suitable display control).

In some embodiments, front panel 304 can be a touch surface. Forexample, head-mounted device 300 can be connected to a portableelectronic device, and a user may navigate the screen of the portableelectronic device by moving one or more fingers across front panel 304.

Head-mounted device 300 may associate user inputs on front panel 304with portable electronic device functions using any suitable approach.For example, the user may be looking at the screen of the portableelectronic device while wearing head-mounted device 300. When the userslides a finger across front panel 304, head-mounted device 300 cantranslate this movement to a movement of a cursor or control on thescreen of the portable electronic device.

The movement of the user's fingers and the effect on the screen can berelated using any suitable approach. For example, this association canbe a one-to-one association. As another example, this association can bescaled (e.g., smaller or larger). A smaller association may be possiblewhen a movement on front panel 304 corresponds to a smaller movement onthe screen. Such an association can be useful, for example, when a useris navigating a list. A larger association may be possible when amovement on front panel 304 corresponds to a larger movement on thescreen. Such an association can be useful, for example, when a user isflipping through pictures in a slideshow.

In some embodiments, head-mounted device 300 can include connectors orjacks 320, which can be one or more external connectors that can be usedto connect to other external devices or systems (data and/or power). Inone embodiment, head-mounted device 300 may include a secondaryaudio/video connector 320 when the portable electronic device issituated inside the head-mounted device. Alternatively, head-mounteddevice 300 may include a hole for allowing access to the audio/videoconnector on the portable electronic device.

Head mounted device 300 may include a camera region 306. Camera region306 can be positioned proximate front panel 304. The camera region mayrepresent a camera that is integrated with the head mounted deviceand/or it may represent an opening for a camera. An integrated cameramay be used in place of or in conjunction with a camera on the portableelectronic device. For example, a camera on portable electronic device202 (FIG. 2) may not be able to capture the user's surroundings whencoupled to the head-mounted device. As another example, camera 306 canbe used if the portable electronic device does not include a camera ofits own (e.g., iPod Touch™ available from Apple Inc. of Cupertino,Calif.).

In cases where the portable electronic device includes a camera, andthere is a desire to reduce redundancies (e.g., thereby reducing weight,complexity and cost), front panel 304 can include a camera hole, whichcan be aligned with a camera of the portable electronic device when theportable electronic device is situated inside head-mounted device 300.The camera hole can allow the camera on the portable electronic deviceto capture image based content of the user's surroundings. While a holeis not shown, it should be appreciated that it can be placed at anylocation on the front panel.

In some embodiments, camera(s) associated with camera region 306 can beused to capture image based content (e.g., images and video) of theuser's external surroundings. For example, camera(s) can be used whenhead-mounted device 300 is worn on the user's head to provide imagebased content to the user. This may allow the user to see what he wouldnot normally be able to because his vision is blocked or impaired bywearing head-mounted device 300. This may advantageously allow the userto wear head-mounted device 300 for extended periods of time.

Head mounted device 300 may include speakers 308. Speakers 308 can belocated at various locations on head-mounted device 300 to enhance theuser's viewing experience. For example, speakers 308 can be placedaround some or all of the periphery (e.g., sides, top, and/or bottom) offrame 302. As another example, speakers can be integrated into strap310, which can be located at the user's ear level. As still anotherexample, speakers 308 can be placed on eyeglass temples 312, which canfit over or behind the user's ears. Speakers 308 can include a varietyof different types of speakers (e.g., mini speakers, piezo electricspeakers, and the like), and/or haptic devices.

Haptic devices (e.g., buzzers, or vibrators) can work alone or incombination with speakers. In some cases, the speakers may serve ashaptic components. Similarly to the speakers, haptics can be placedaround some or all of the periphery (e.g., sides, top, and/or bottom) offrame 302. As another example, haptics can be integrated into strap 310,which can be located at the user's ear level. As still another example,speakers 308 can be placed on eyeglass temples 312, which can fit overor behind the user's ears.

Haptic devices can interface with the user through the sense of touch byapplying mechanical stimulations (e.g., forces, vibrations, andmotions). For example, while a user is watching image based content,haptic devices can be configured to provide an enhanced surround soundexperience by providing impulses corresponding to events in the imagebased content. As an illustrative example, the user may be watching amovie that shows an airplane flying on the left of the screen. Hapticdevices can produce vibrations that simulate the effect (e.g., soundeffect, shock wave, or any combination thereof) of the airplane. Forexample, a series vibration may be provided along the left temple fromfront to back to simulate the airplane flying to the left and rear ofthe user. Speakers can also be used in this manner.

Head mounted device may include a variety of ports 314. When portableelectronic device is coupled to head-mounted device 300, ports 314 canprovide a path for signals (e.g., audio signals) to travel between theportable electronic device and head-mounted device 300. In someembodiments, ports 314 can provide a simple path for signals to travelto the user. For example, ports 314 can be a hole or simple circuitry(e.g., an amplifier) that is directly aligned with a component on theportable electronic device. As a result, sound can travel directly fromthe portable electronic device to the user. In some embodiments, ports314 can provide a complex path for signals, in which signals areredirected to a different location on the head-mounted device. Forexample, head-mounted device 300 can include circuitry that can redirectaudio signals from speakers on the portable electronic device to variouslocations on the head-mounted device (e.g., audio signals can beredirected to speakers 308 and earphones 316). As another example,head-mounted device 300 can include an acoustical port for allowingaccess to a microphone on the portable electronic device. Alternatively,the head-mounted device may include its own microphone.

In some embodiments, head-mounted device 300 can further include one ormore electronic subassemblies 324. The electronic subassemblies 324 canbe configured to implement any suitable functionality provided byhead-mounted device 300. For example, the electronic subassemblies 324can include memory (e.g., memory 1318 of FIG. 13), sensors (e.g.,sensors 1324 of FIG. 13), processors (e.g., processor 1310 of FIG. 13),communications circuitry (e.g., communications circuitry 1328 of FIG.13), battery (e.g., battery 1320 of FIG. 13), or any other suitablecomponent. The one or more subassemblies 324 may be placed at variouslocations within or outside of the head-mounted device 300. For example,the electronic subassemblies may be disposed at internal spaces definedby the main body 303 and possibly the front and rear panels 304 and 305(without interfering with the internal space provided for the portableelectronic device). In one example, they are placed at the lowersections on the right and left of the nose support region of the mainbody 303. This section would otherwise not be used because of theconfiguration of the portable electronic device (e.g., substantiallyrectangular). Additionally or alternatively, the main body 303 and/orthe front panel 304 may form enclosed portions that extend outwardlythereby forming internal spaces for placing the electronicsubassemblies.

Remote control 322 can be connected to head-mounted device 300 or theportable electronic device using any suitable approach. For example,remote control 322 can be a wired device that is plugged into aconnector in connectors 320. As another example, remote control 322 canbe a wireless device that can transmit commands to the portableelectronic device and head-mounted device 300 via a wirelesscommunications protocol (e.g., Wi-Fi, infrared, Bluetooth™, or anycombination thereof). As still yet another example, remote control 322can be a device that is capable of both wired and wirelesscommunications. The user may use remote control 322 to navigate theportable electronic device and to control the display, volume, andplayback options on the portable electronic device. Remote control 322can include any remote control device, such as, for example, the AppleRemote available from Apple Inc. of Cupertino, Calif.

FIG. 4 shows a configuration for sliding a portable electronic device400 into a head-mounted device 300 in accordance with one embodiment ofthe invention. Portable electronic device 400 and head-mounted device300 can correspond to any of those previously described.

In the implementation shown in FIG. 4, the portable electronic device400 is slid through the slot 340 and into the cavity 332. The slot 340may be situated opposite a connector 342 such that as the portableelectronic device 400 is slid into the cavity 332, a connector 402 ofthe portable electronic device 400 can begin to mate with connector 342and can completely mate upon final insertion of the portable electronicdevice 400 into the cavity 332. In the illustrated embodiment, slot 340may be configured to receive the bottom end of the portable electronicdevice 400. During insertion, the walls of the cavity 332 help guide theportable electronic device 400 into place within the cavity 332. Inaddition, the walls help guide the connector 402 of the portableelectronic device 400 to connect with the corresponding connector 342disposed within the cavity 332.

The connectors 402 and 342 can allow data and/or power communicationsbetween head-mounted device 300 and portable electronic device 400. Theconnectors 402 and 342 may be widely varied, such as, for example, lowprofile connectors, USB, FireWire, or a 30 pin connector used in iPods™and iPhones™ manufactured by Apple Inc. of Cupertino, Calif. In somecases, the cavity/connector combination may generally define a dockingstation for portable electronic device 400. As shown in FIG. 4, theconnectors 402 and 342 may be male/female. For example, the portableelectronic device may include a female connector while the head-mounteddevice may include a male connector. The male connector is inserted intothe female connector when the devices are coupled together.

Because connector 402 may be blocked once portable electronic device 400is coupled to head-mounted device 300, portable electronic device 400may be prevented from interfacing with other devices. To accommodateother devices, head-mounted device 300 can include an adaptor 343 thatincludes connector 344 on the outside of the head-mounted device 300. Asa result, when another device (e.g., a battery) is plugged intoconnector 344, this device can interface with portable electronic device400.

After coupling the portable electronic device 400 to the head-mounteddevice 300, the protocol under which devices communicate may be widelyvaried. Any suitable communication protocol may be used, such as, forexample, a master/slave communication protocol, server/clientcommunication protocol, peer/peer communication protocol, or anycombination thereof. For example, using a master/slave communicationprotocol, one of the devices, the master device, controls the otherdevice, the slave device. For instance, the portable electronic device400 may become a slave to the head-mounted device 300 such that thehead-mounted device controls the operation of the portable electronicdevice once they are coupled. Alternatively, the head-mounted device 300can serve as a slave of the portable electronic device 400 by simplyimplementing actions based on controls from the portable electronicdevice 400. As another example, using a client/server communicationprotocol, a server program, operating on either portable electronicdevice 400 or head-mounted device 300, responds to requests from aclient program. As yet another example, using a peer to peercommunication protocol, either of the two devices can initiate acommunication session.

Generally, the communication session can begin when the two devices arecoupled together and powered up. In some cases, the devices may switchimmediately into a head-mounted mode once connected. In the head-mountedmode, the size and possibly the resolution of one or more image frameson the screen may be adjusted for viewing at close proximity. Inaddition, some features can be turned on/off while the settings of otherfeatures may be reconfigured differently from normal portable electronicdevice use. For example, input devices, output devices, sensors, andother electrical systems can be activated or deactivated based on thedefault settings.

In one embodiment, the user may be provided with an option to set up thehead-mounted display system manually. For example, the user may beprompted with a control menu for setting up the head-mounted displaysystem when the devices are coupled together. As another example, theuser may be provided with an option to manually set the distance betweentwo image frames that are displayed on the screen (one for each eye).For example, the user may select a control (e.g., rotate a knob) toalign the two image frames with his eyes.

Alternatively, if adjustments are difficult for the user, head-mounteddevice 300 and/or portable electronic device 400 may include mechanismsfor automatically configuring the image frame location and size based ondefault settings. As should be appreciated, each user's eyes areoriented differently. For example, some eyes are located close togetherwhile others are more spread out. Thus, either device may includesensors for detecting the distance to the eyes and the position of theeyes. After determining the optimal viewing positions and size of thedisplayed image frames based on the detected distance and position, thehead-mounted display system can adjust the viewing positions. By way ofexample, the portable electronic device and/or the head-mounted devicemay include cameras that can reference where the eyes are locatedrelative to the head-mounted device.

The resolution of the displayed image frames can also be adjusted in asimilar manner. However, because each user's eyes focus differently, itmay be beneficial to allow the user to manually adjust the resolution asthis is a more difficult measurement to make.

In one embodiment, the head-mounted display system may include a sensingmechanism for alerting the portable electronic device 400 that thedevice has been coupled to the head-mounted device 300. As a result,portable electronic device 400 can switch to the head-mounted mode. Byway of example, the sensing mechanism may be an electrical connection, asensor such as a proximity sensor or IR detector, and/or the like. Thesensing mechanism may be used instead of or in combination with thecommunication interface to assist the devices into switching to thehead-mounted mode.

In addition, as a result of operatively coupling the devices throughwired and/or wireless interfaces, features of the head-mounted device300 may be extended to the portable electronic device 400, and featuresof the portable electronic device 400 may be extended to thehead-mounted device 300. Furthermore, redundant features may be removedfrom head-mounted device 300, and enhancement features may be added tohead-mounted device 400.

FIG. 5 shows a side view of a head-mounted device 500 coupled to aportable electronic device 502 in accordance with an alternateembodiment of the invention. In this embodiment, a portion of theportable electronic device 502 is configured to protrude out of the HMD500. It can be configured to protrude out of any side including right,left, bottom and top. In the illustrated embodiment, the portableelectronic device 502 protrudes out of the top of the HMD 500 (i.e., HMD500 includes a top slot and a cavity 504 only configured for enclosing aportion of the portable electronic device 502). In this type ofconfiguration, the portion 507 of the display screen 506 positionedbehind an opening 508 is utilized for displaying image areas. This isgenerally not an issue as the size of the image areas is typically muchsmaller than the overall screen size (i.e., reduced or shrunk so thatthey can be easily viewed).

FIGS. 6A and 6B show a top view of a head-mounted device 600 coupled toa portable electronic device 602 in accordance with one embodiment ofthe invention. This embodiment is similar to the embodiment shown inFIGS. 3 and 4 except that the head mounted device further includes anoptical subassembly 604 for helping to provide a better viewingexperience. Optical subassembly 604 may be disposed between the screen606 of the portable electronic device 602 and the user's eyes (e.g.,overlaid over at least the viewing portion of the screen). In somecases, the optical subassembly 604 can be located in the region of anopening (e.g., opening 334 of FIG. 3). The optical subassembly 604 may,for example, be disposed behind, within or in front of the opening. Theoptical subassembly 604 may be supported by a main body (e.g., main body303 of FIG. 3) and/or a back panel (e.g., back plane 305 of FIG. 3) ofthe head mounted device 600. For example, the optical subassembly 604may be attached to the main body via any suitable means including, forexample, screws, adhesives, clips, snaps, and the like.

The optical subassembly 604 may be widely varied. For example, theoptical subassembly 604 may have a planar configuration. In some cases,the optical subassembly may simply be a transparent window that projectsthe image based content to the user. The window can be made from anysuitable clear material, such as, for example, polycarbonate or glass.In other cases, however, the optical subassembly 604 may be a morecomplicated system of optical components that may be configured toadjust image based content playing on the portable electronic device sothat the image based content can be viewed at close proximity (e.g.,help focus the user's eyes on the image based content being displayed onthe screen of the portable electronic device)

The optical subassembly 604 may include various optical components thatmay be static or dynamic components depending on the needs of thesystem. The optical components may include, for example, lenses, lightguides, light sources, mirrors, diffusers, optics, lasers or any othersuitable optical component. Specific examples of optical subassembliesmay include aspherical and diffractive optical arrangements. Asphericaloptical arrangements may include various types of prism and/or lensarrangements (e.g., a coaxial arrangement of lenses, a coaxialarrangement of a set of lenses arranged along a common axis, a concavemirror arrangement, and a free shaped prism arrangement). Diffractiveoptical arrangements may include various types of arrangements that bendand spread light (e.g., a light-guided optical element, or a binocularlight-guided optical element, both of which can be made of planartransparent glass or plastic). A more detailed discussion on processingand displaying image frames using optics in a head-mounted device can befound, for example, in Tang, U.S. Published Patent Application No.2008/0088529, published Apr. 17, 2008, entitled “Enhanced Image Displayin Head-Mounted Displays,” which is incorporated by reference in itsentirety, and in Tang et al., U.S. Published Patent Application No.2007/0201737, published Aug. 30, 2007, entitled “Peripheral Treatmentfor Head-Mounted Displays,” which is also incorporated by reference inits entirety. A discussion on the use of laser and optics to processimage frames on head-mounted devices can be found in Tang, U.S.Published Patent Application No. 2008/0088937, published Apr. 17, 2008,entitled “Head Mounted Display System,” which is hereby incorporated byreference in its entirety.

The optical subassembly 604 may include one or more optical interfacesthat may be operative to adjust or modify displayed image frames basedon any suitable criteria (e.g., as standalone modules or using a circuitboard). For example, the optical subassembly may include a singleoptical interface (as shown in FIG. 6A). In one implementation, theoptical subassembly 604 can be formed as a panel that overlays theopening. The panel may be curvilinear and/or rectilinear. For example,the optical subassembly 604 may be a thin flat panel that can be easilycarried by the main body and easily supported on a user's head. Asanother example, the optical subassembly may include several opticalinterfaces 605 and 606 (e.g., one for each eye as shown in FIG. 6B).

The optical interface(s) may also be adjustable. For example, theoptical interface(s) may be operative to resize or otherwise modifyimage frame(s) such that the image frame(s) appears to be displayed at agreater distance than the distance between the optical interface(s) andthe user's eyes (e.g., the image frame(s) appears to be viewed from 4meters and not 20 mm). As another example, optical interface(s) may beoperative to generate displays that correspond to standard definition orhigh definition displays.

In some embodiments, left and right optical interfaces 605 and 606 maygenerate and display different image frames associated with the imagebased content. For example, left and right optical interfaces 605 and606 (e.g., upon receiving appropriate instructions from the head-mounteddevice or portable electronic device) may offset the left and rightimage frames so that the user is given the illusion of viewing imagebased content in three dimensions. The left and right optical interfaces605 and 606 may identify the appropriate left and right image framesusing any suitable approach, including, for example, receiving separateleft and right image frames from the imaged based content source.Alternatively, the left and right optical interfaces 605 and 606 maygenerate the left and right image frames from a single image frameprovided by the image based content source.

In one implementation, left and right optical interfaces 605 and 606 mayadjust the displayed image frames to account for limitations orcorrections (e.g., by glasses or contact lens) of the user's eyesight.For example, if a user is myopic in one eye, the optical interfaceassociated with that eye may modify the image frame displayed to correctfor the user's myopia. The head-mounted display system may determine thecorrection required for each eye using any suitable approach. Forexample, the user may enter a glasses or contact lens prescription thatindicates the required correction. As another example, each opticalinterface may automatically analyze the user's eyes, and determine theadjustment needed based on the analysis. This may allow users whonormally wear glasses to use the head-mounted display system withouttheir glasses.

In some embodiments, if the portable electronic device 602 is operativeto adjust image based content for viewing at close proximity, thehead-mounted device 600 may not include an optical subassembly. In suchembodiments, the display screen of head-mounted device can serve as aclear cover for the screen of the portable electronic device. Thedisplay capabilities of the head-mounted device and the portableelectronic device will be described in more detail in connection withFIGS. 11A-11F.

FIG. 7 shows a side view of a head-mounted device coupled to a portableelectronic device in accordance with one embodiment of the invention.Head-mounted device 700 can include eyeglass temples 702, which allowsthe user to wear head-mounted display system 700 by putting eyeglasstemples 702 over his ears. Thus, instead of sliding the portableelectronic device 704 into the head-mounted device 700, as shown in theprevious embodiment in FIG. 4, the portable electronic device can becoupled to the head-mounted device using clips.

In some embodiments, head-mounted device 700 can include top clip 706and bottom clip 708 to hold portable electronic device 704 in place infront of the user's eyes. In the embodiment shown in FIG. 7, the screenof portable electronic device 704 may face the user's eyes.

Clips 706 and 708 can be formed from any suitable material, such as forexample plastic, rubber, and foam. In some embodiments, clips 706 and708 can follow the contours of portable electronic device 704. The usermay place portable electronic device 704 into head-mounted device 700 bysnapping portable electronic device 704 into bottom clip 708. The usercan then rotate portable electronic device 704 until the rest of thedevice fits into clips 706 and 708 (e.g., following an elasticdeformation of at least one of clips 706 and 708). In some embodiments,the user may slide portable electronic device 704 into clips 706 and708.

A side cover (not shown) located on either side of head-mounted device700 may include a complementary connector that can be coupled to aconnector of portable electronic device 704 (e.g., connector 512 of FIG.5). For example, the side cover may be a door that swings from an openposition allowing portable electronic device 704 to be inserted. Onceportable electronic device 704 has been inserted, the side cover mayclose and the complementary connector can be coupled to portableelectronic device 704.

The head-mounted device can include any suitable feature for improvingthe user's comfort or ease of use when the portable electronic device iscoupled to the head-mounted device. FIGS. 8A-8D show illustrativefeatures for exemplary head-mounted devices. FIG. 8A shows ahead-mounted device that includes foam 802 on at least a lower portionof the device. Foam 802 can be made from any foam or material that candeform or substantially comply with the user's face (e.g., nose) thusimproving the user's comfort. In some cases, the foam 802 is placed atthe location where the frame engages the nose (e.g., nose cut out). Inother cases, the foam is placed continuously or selectively across theentire bottom edge that engages the nose and face. Still further, thefoam may be placed continuously or selectively across the entire edge ofthe frame that engages the nose and face (upper, side and lowerportions). In fact, in some implementations, because the material isdeformable, the bottom surface of the head-mounted device can be flatwhen the device is not being worn (e.g., no nose cut out).

Foam 802 can be used to prevent ambient light from entering between theuser's face and the head-mounted device (e.g., provides a seal betweenthe frame and the user's face). Additionally, foam 802 can be used toreduce the load on the user's nose because the portable electronicdevice can be relatively heavy. In some cases, foam 802 can serve toincrease a user's comfort with the head-mounted device by helping tocenter the head-mounted device on the user's face.

Alternatively or additionally, the head-mounted device may include ashroud (not shown) that helps enclose the viewing experience. The shroudmay, for example, be one or more shaped panels that fill and/or coverthe air gaps normally found between the frame and the user's face. Infact, the deformable material may be applied to the shroud.

In some embodiments, the head-mounted device can serve a secondarypurpose for the portable electronic device. For example, FIG. 8B shows ahead-mounted device that has adjustable eyeglass temples. In theembodiment shown in FIG. 8B, eyeglass temples 804 can slide or fold intothe frame of the head-mounted device. As a result, the head-mounteddevice can serve as a case for the portable electronic device. In thisembodiment, the frame may include one or more slots for receiving bothof the eyeglass temples. The ends of the eyeglass temples may beconfigured to prevent the temples from exiting the slot, and, at thesame time, allow the temples to pivot when the user wants to wear thedevice. By way of example, the ends of the eyeglass temples may includea pivot pin that is sized larger than the slot opening.

FIG. 8C shows a head-mounted device that has an adjustable frame. In theembodiment shown in FIG. 8C, the frame of the head-mounted device canrotate toward the user. For example, the head-mounted device can havepivot bearing 806 (e.g., hinges) connecting the frame and eyeglasstemples 808. Pivot bearing 806 can allow the frame to rotate aroundeyeglass temples 808. As a result, the head-mounted device can serve asa display stand for the portable electronic device if the user decidesnot to wear the head-mounted device.

FIG. 8D shows a head-mounted device that has eyeglass temples that canrotate. In the embodiment shown in FIG. 8D, eyeglass temples 810 canrotate around the frame of the head-mounted device so that thehead-mounted device can serve as a display stand. For example, thehead-mounted device can have pivot bearing 812 (e.g., hinges) connectingthe frame and eyeglass temples 810. Pivot bearing 812 can allow eyeglasstemples 810 to rotate around the frame.

The head-mounted display system may display image frames in variousmodes depending on the manner in which it is being used. For example,when the head-mounted device is being worn, the system may adjust thescreen for viewing at close proximity (e.g., by splitting the screeninto one or more smaller image frames). On the other hand, when thehead-mounted device is used as a display stand, the system may provide asingle large image frame by maximizing the screen dimensions.

FIGS. 9A-9C illustrate a head-mounted device 900 in accordance withanother embodiment of the present invention. The head-mounted device 900may, for example, correspond to any of those embodiments previouslydescribed. In one embodiment, the head-mounted device 900 does notinclude a primary display and instead relies on the display of aportable electronic device to provide image-based content to a user. Thehead-mounted device 900 can include a main body 902 to which a frontpanel 904 and a rear panel 906 are attached to form the frame of thehead-mounted device 900. The head-mounted device can also includeeyeglass temples 910 for helping support the head-mounted device on theuser's head. The eyeglass temples 910 are moveably attached to the frame902. FIG. 9A is a top elevation view of the entire assembly. FIG. 9B isa front elevation view with the front panel removed. FIG. 9C is a rearelevation view of the front panel.

As shown, the main body 902 defines a cavity 912 and a slot opening 913for receiving a substantially flat small form factor portable electronicdevice. The front panel 904, main body 902 and rear panel 906 cooperateto form various enclosed spaces for various electronic components andwiring of the head-mounted device 900. As shown, these members may format least a front space 918 and a side space 920.

The main body 902 also includes one or more openings 914 through whichimage based content is displayed when a portable electronic device isdisposed within the cavity 912 (e.g., the screen of the portableelectronic device is placed behind the opening). The rear panel 906and/or the main body 902 can be configured to carry an opticalsubassembly 916 placed in front of the opening 914. The opticalsubassembly may help in viewing the image based content being displayedby a portable electronic device.

As shown in FIG. 9B, the front space 918 can be broken up into a varietyof zones. In this particular implementation, the front space includesthree zones 918A, 918B and 918C. Each of these zones can include avariety of electrical and mechanical components including but notlimited to batteries, processors, circuit boards, cooling systems,memory, sensors, I/O, and the like. In the illustrated embodiment, thespaces include a touch sensor 924, camera 926, battery 928 and coolingsystem 930.

As further shown in FIG. 9B, the side space can also include a varietyof electrical and mechanical components as mentioned above. In theillustrated embodiment, the side space includes a main circuit board 931that can be broken up into multiple boards if needed. The main circuitboard 931 is configured to carry a connector 932 that protrudes into thecavity 912 opposite the slot opening 913. The main circuit board 931 canalso carry a variety of controls 933 that protrude outside the side wallof the main body 902. The main circuit board 931 can also carry avariety of chips including processors, memory (flash) and the like. Themain circuit board 931 may be operatively coupled to each of theelectrical components via wires, traces and/or flex circuits dependingon the needs of the system (these are shown by solid dark lines for easeof illustration)

As shown in FIGS. 9A and 9B, the temples 910 are coupled to the side ofthe main body 902. Although the location may vary, in the illustratedembodiment, the temples are connected towards the center of the frame inorder to better align the optical sub assembly with a user's eyes. Theconnection point between the ends of the temples 910 and the main body902 may include a through hole for passing wires, traces and/or flexcircuits. These elements connect to various electrical componentscarried by the temples 910. The electrical components may be widelyvaried. In the illustrated embodiment, the temples include a series ofspaced apart haptic mechanisms 940. The temples 910 can also include atouch sensor 942, which can be used as a control (e.g., controlling thevolume by sliding a finger across touch sensor 942). The temples canalso include a speaker 944 for each ear. The speakers may be housedwithin earpieces that fit inside the user's ear. Alternatively, thespeakers may include a grill proximate the location of the ear.

As shown in FIG. 9C, the front panel includes a touch sensing layer 924on its inside surface. The touch sensing layer 924 may be a capacitivetouch sensing layer. In this particular embodiment, the front panel maybe formed from a dielectric material such as plastic. The touch sensinglayer 924 may, for example, be integral with or embedded on a surfacelayer such as a flex circuit or printed material. The touch sensinglayer 924 may couple to the main circuit board 931 through wires, tracesand/or flex circuits. It may, for example, include a connector/wire forinterfacing with the main circuit board 931. The front panel can alsoinclude a vent region 950. The vent region 950 may be a series of ports952 through which heat may be distributed as, for example, via a coolingmechanism such as a fan.

The front panel, main body and rear panel may couple via any suitabletechnique including but not limited to adhesives, fasteners, snaps,clips, locks, or any combination thereof.

FIGS. 10A and 10B illustrate an exemplary portable electronic devicethat may be used in any of the head-mounted devices described herein. Itis particularly applicable to the embodiment shown in FIGS. 3-5. In thisembodiment, the portable electronic device is a handheld electronicdevice. Thus, it has a small form factor that can be easily managed by ahead-mounted device.

Portable electronic device 1000 can include receiver 1002, which caninclude any component operative to receive electric signals and convertthe signals into sound for a user. For example, a speaker included inreceiver 1002 can be used to broadcast sound to the user.

A head-mounted device may include an acoustical component thatinterfaces with the receiver. A head-mounted device may also include aredundant receiver on one of its external surfaces or in an earphone.

Portable electronic device 1000 can include connector 1004, which can beplugged into a complementary connector on another device. Any suitabledevice can be coupled to portable electronic device 1000, such as, forexample, an accessory device, host device, external power source, or anycombination thereof. A host device can be, for example, a desktop orlaptop computer or data server from which portable electronic device1000 can receive image based content files. Persons skilled in the artwill appreciate that connector 1004 can be any suitable connector, suchas USB, FireWire and/or 30-pin connector available from Apple Inc. ofCupertino, Calif. A head-mounted device may include a complimentaryconnector such that the head-mounted device and the portable electronicdevice can be operatively coupled when the portable electronic device ismounted relative to the head-mounted device.

Portable electronic device 1000 can include screen 1006 for displayingimage based content to a user. Screen 1006 can enable portableelectronic device 1000 to play back the video portion of media, displayimages or command menus, or serve any other suitable display function.Although screen size can be widely varied, in the illustratedembodiment, the screen size is configured to fill a majority of thefront surface of the portable electronic device (e.g., substantiallyfull screen). In one embodiment, the screen is a high resolution screen.This may enable head-mounted device viewing. In one example, the screenmay operate in a first resolution for portable electronic device viewing(e.g., normal viewing mode) and a second resolution for head-mounteddevice viewing. The second resolution for head-mounted device viewingmay be higher than the first resolution for portable electronic deviceviewing. For example, head-mounted device viewing may operate in highresolution while portable electronic device viewing may operate inmedium resolution. In one particular embodiment, screen 1006 is utilizedas the primary screen when the portable electronic device is coupled tothe head-mounted device.

In some cases, screen 1006 may be a multi-touch screen that may functionas both an integrated display screen and as a user input device. Someembodiments of multi-touch screens are discussed in more detail incommonly assigned Westerman et al., U.S. Pat. No. 6,323,846, issued Nov.27, 2001, entitled “Method and Apparatus for Integrating Manual Input,”which is incorporated by reference herein in its entirety.

Portable electronic device 1000 can include button 1008, which can beused in conjunction with screen 1006. Button 1008 can be used, forexample, to turn on screen 1006 or return to a display screen on screen1006 (e.g., a home screen or springboard) Persons skilled in the artwill appreciate that additional buttons and/or other user input devicesmay be used without departing from the spirit of the present invention.

Portable electronic device 1000 can include camera 1010, which cancapture image based content (e.g., images or video) of the outsideworld. In some embodiments, when camera 1010 is activated, the imagebased content can be displayed on screen 1006. The user may take apicture or capture a video using any suitable approach, including, forexample, by pressing button 1008 or selecting an option on screen 1006.

Portable electronic device 1000 can include speaker 1018, which can beused to provide the user with audio when headphones 1014 are notconnected to portable electronic device 1000. For example, in hands-freeoperation (e.g., speakerphone mode), speaker 1018 can allow the user tohear the voice of the other speaker during a telephone conversation. Asanother example, speaker 1018 can allow the user to hear the sound ofimage based content during playback.

Portable electronic device 1000 can include microphone 1020, which canbe used to receive sounds when headphones 1014 are not plugged intoportable electronic device 1000.

Portable electronic device 1000 can include audio/video connector 1022for connecting to audio plug 1012. Audio plug 1012 can be used tophysically and electrically couple portable electronic device 1000 andheadphones 1014 together. For example, once audio plug 1012 is insertedinto audio/video connector 1022, audio from portable electronic device1000 can be transmitted to headphones 1014 instead of receiver 1002. Insome embodiments, headphones 1014 can include microphone 1016 forreceiving ambient sound. For example, microphone 1016 can be used toreceive the user's voice while the user is conducting a telephoneconversation while wearing headphones 1014. As another example,microphone 1016 can allow the user to provide voice commands to portableelectronic device 1000.

Although not shown, portable electronic device 1000 can include variousswitches and/or buttons such as, for example, volume switches, holdswitches, mute switches and the like. In addition, portable electronicdevice 1000 can include other input/output devices such as proximitysensors, accelerometers, light sensors and the like. By way of example,a proximity sensor may be used to detect objects that are nearby. Asdiscussed, a head-mounted device may include redundant features such asthese, and/or provide access to these features, for example, viaopenings or holes.

The screen of the portable electronic device may be widely varied and assuch so may the head-mounted device configured for receiving theportable electronic device (as mentioned above). In one implementation,the screen of the portable electronic device may be a full screendisplay that extends across the entire front surface of the device. Inanother implementation, the screen of the portable electronic device maybe a substantially full screen display such that a majority of thescreen fills the front surface of the portable electronic device. In yetanother implementation, the screen may be a partial screen display. Forexample, the partial screen display may only be oriented in the upperhalf of the portable electronic device. As another example, the partialscreen display may be symmetrically centered on the front surface, oralternatively the partial screen display may be offset towards one side.In all of these implementations, the entire screen or portions of thescreen can be covered to provide the desired screen size and/or shapefor enhanced viewing experience.

When the portable electronic device is coupled to the head-mounteddevice, the distance of the screen relative to the user's eyes may bewidely varied. In small form factor head-mounted devices (low profile),the screen of the portable electronic device may be placed fairly closeto the user's eyes. The placement of the screen may be controlled by thesurfaces of the cavity and, more particularly, the walls of the cavity.

Because the screen may be too large when compared to the viewingdistance, the head-mounted display system may be configured to adapt theimage based content displayed on the screen so that it can be easily andproperly viewed (e.g., image based content stored on the head-mounteddevice or the portable electronic device, or image based content from acamera). FIGS. 11A-11F show several examples of creating image frameswithin a display screen in a head-mounted display system when in a HMDmode.

In FIG. 11A, the head-mounted display system has adjusted the size ofthe image based content (image frame 1102) to be less than the size ofthe screen 1104 on portable electronic device 1106. In oneimplementation, the size of image frame 1102 is configured to fill theentire field of view of the user. In another implementation, the size ofimage frame 1102 is configured to be less than the field of view of theuser.

In some cases, the head-mounted display system may also adjust theresolution of the viewable image based content. For instance, theresolution may be increased or decreased relative to normal portableelectronic device viewing to compensate for how close the eyes are tothe display screen of the head-mounted device.

Image frame 1102 can also have different aspect ratios depending on theprimary application of the displayed content. For example, an aspectratio of 4×3 (e.g., similar to a television screen) may be used fortelevision programs, whereas an aspect ratio of 16×9 (e.g., similar to amovie theatre screen) may be used for watching feature films.

It should also be emphasized that the head-mounted display system is notlimited to displaying one image frame. In some cases, it may bebeneficial to provide multiple image frames (e.g., two image frames) toimplement stereoscopic imaging (see, for example, FIGS. 11B-11F).Stereoscopic imaging attempts to create depth to the image frames bysimulating the angular difference between the image frames viewed byeach eye when looking at an object, due to the different positions ofthe eyes. This angular difference is one of the key parameters the humanbrain uses in processing images to create depth perception or distancein vision.

In one example, a single source image frame, which can be provided bythe head-mounted device, portable electronic device or an externalmedium, is processed (either by the head-mounted device or the portableelectronic device) to generate a left image frame and a right imageframe for viewing. As a result, the left and right image frames may bedisplayed at two locations within the display screen. The timing orimage characteristics of these two image frames relative to one anothermay be varied to provide an enhanced viewing effect and an immersiveexperience for the user. For example, one image frame can be offset fromthe other image frame so that the user is given the illusion of viewingimage based content in three dimensions. Persons skilled in the art willappreciate that the head-mounted display system can identify theappropriate left and right image frames using any suitable approach. Amore detailed discussion of adjusting image frames on a head-mounteddevice can be found in Prest et al., U.S. patent application Ser. No.12/114,499, Attorney Docket No. P5222US1, filed on May 2, 2008, entitled“Head-Mounted Display,” while is hereby incorporated by reference in itsentirety.

FIG. 11B shows a back view of ahead-mounted display system for providingtwo image frames in accordance with one embodiment of the invention.Head-mounted display system 1108 can provide a primary image frame onthe display screen. The primary image frame can include left image frame1110 and right image frame 1112 opposite the user's left and right eye,respectively.

In some embodiments, a portable electronic device that is coupled tohead-mounted device 1114 can create or generate distinct left and rightimage frames 1110 and 1112 for display by the head-mounted device. Theportable electronic device can separate or distinguish left and rightimage frames 1110 and 1112 using any suitable approach. For example, theportable electronic device can display left and right image frames 1110and 1112 adjacent on the screen of the portable electronic device (e.g.,with no separating element on the screen), such that limitations to theuser's field of view (e.g., due to the user's nose) may allow the userto distinguish the image frames. As another example, the portableelectronic device can separate left image frame 1110 and right imageframe 1112 by displaying a black overlay over an area between oradjacent the periphery of the two image frames (e.g., area 1116). Thismay enhance a user's experience by giving the user the impression ofbeing in a theater or other optimal environment for viewing image basedcontent. As yet another example, the head-mounted device can include ablack stencil defining image frames in the display screen for left imageframe 1110 and right image frame 1112. Persons skilled in the art willappreciate that any suitable overlay (e.g., theater walls, textualoverlay, or outline of other viewers) can be used in area 1116 toenhance the user's viewing experience. A more detailed discussion on theuse of overlays for head-mounted devices can be found, for example, inTang, U.S. patent application Ser. No. 12/072,762, filed on Feb. 27,2008, entitled “Adjusting Media Display in a Personal Display SystemBased on Perspective,” which is incorporated by reference in itsentirety.

In most cases, left and right image frames 1110 and 1112 and image frame1102 of FIG. 11A are typically centered within the primary image frame.However, in some cases, it may be desirable to adjust the position ofthe image frames within the primary image frame in order to better alignthe image frames to the user's eyes (align or center the displayed imageframes relative to where the eyes should be or to where the eyesactually are). As should be appreciated, each user's eyes are orienteddifferently. For example, some eyes are located close together whileothers are more spread out.

The alignment may be based on a particular location relative to thehead-mounted device and/or based on the position of the user's eyesrelative to the screen. For example, the image frames may be centeredrelative to a nose support of the frame. As another example, eitherdevice (the head-mounted device or the portable electronic device) mayinclude sensors (e.g., optical sensors) for detecting the distance tothe user's eyes and the location of the user's eyes. By way of example,the portable electronic device and/or the head-mounted device mayinclude cameras that can reference where the user's eyes are locatedrelative to the head-mounted device. Alternatively, the portableelectronic device may become a slave to the head-mounted device suchthat the head-mounted device controls the operation of the portableelectronic device once it is inserted into the head-mounted device.

The two image frames may be adjusted in any suitable manner. Forexample, as shown by arrows in FIG. 11C, the position of the two imageframes may be adjusted together. As another example, as shown by arrowsin FIG. 11D, the position of the two image frames may be adjustedrelative to one another. For instance, one may be adjusted upward whilethe other may be adjusted to the right.

As yet another example, only one of the two image frames may be adjusted(e.g., if the portable electronic device is coupled to the head-mounteddevice such that the screen of the portable electronic device is alignedwith one of the user's eyes). FIG. 11E shows an example where only aportion of the portable electronic device is received by thehead-mounted device. In this embodiment, the lower portion of the screenis utilized to show the two image frames within the primary image frame.Furthermore, although not shown, the orientation of each image frame mayalso be adjusted together or separately (rotation).

If the screen of the portable electronic device is larger than theseparation between the user's eyes, the portable electronic device candirectly display the image based content to the user without usingadditional components on the head-mounted device. For example, a sensoron the portable electronic device can detect the location of the user'seyes. Based on the distance, the portable electronic device can alignleft image frame 1110 with the left eye of the user and right imageframe 1112 with the right eye of the user.

If instead the screen of the portable electronic device is smaller thanthe separation between the user's eyes, an optical subassembly (e.g.,optical subassembly 604 of FIG. 6) can be used to adjust the imageframes provided by the portable electronic device for viewing at closeproximity.

The optical subassembly, which can be located between the user's eyesand the screen of the portable electronic device, can receive one ormore image frames from the portable electronic device. For example, theoptical subassembly can receive left image frame 1110 and right imageframe 1112 from the portable electronic device. A sensor on thehead-mounted device can then detect the interpupillary distance betweenthe user's eyes. Based on the interpupillary distance, the opticalsubassembly can shift at least one of left image frame 1110 and rightimage frame 1112 to align with the user's eyes.

In some embodiments, the optical subassembly can take a single imagereceived from the portable electronic device and produce two imageframes from the single image frame. For example, the optical subassemblycan include two optical interfaces (e.g., optical interfaces 605 and 606of FIG. 6), which are each capable of generating and displaying an imageframe (left image frame 1110 or right image frame 1112) from a singleimage frame. Persons skilled in the art will appreciate that the opticalinterfaces can be configured to properly align left image frame 1110 andright image frame 1112 with the user's eyes. Persons skilled in the artwill also appreciate that the optical interfaces can produce anysuitable overlay for area 1116.

Moreover, the resolution of the left and right image frames 1110 and1112 can also be adjusted. For example, the resolution of each imageframe may be increased to compensate for how close the eyes are to thedisplay screen. Additionally, left and right image frames 1110 and 1112can also have different aspect ratios depending on the primaryapplication of the displayed content.

In some embodiments, the display screen (e.g., the display screen of thehead-mounted device or the screen of the portable electronic device) canbe utilized to extend the view of the displayed image based content. Forexample, if the display screen is larger than the reduced sized imageframe(s), portions of the display screen surrounding the image frame(s)may be used to extend the feel of the displayed image frame(s) toproduce secondary effects (e.g., enhanced viewing effects). The extendedportions may be colors associated with the edge of the displayed imageframe.

The secondary effect may be continuous or segmented (e.g., the effectmay extend from all edges or just corners). The characteristics of thesecondary effect, including speed, can also be adjusted to producedistinct effects. Alternatively or additionally, the image frames may bemade larger and smaller to produce a viewing effect. Other examplesinclude shaking, shifting, etc. See, for example, FIG. 11F which shows adisplay screen 1118 broken up into a set of two primary image regions1120 (e.g., showing the image based content) and a set of secondaryvisual effect regions 1122. The secondary visual effect regions 1122surround at least a portion of each of the primary image regions, andextend and provide enhancements to the images displayed in the primaryimage regions.

In some embodiments, the head-mounted display system can operate in aninternal viewing mode in which stored image based content can beprovided for the user's viewing pleasure. The internal viewing mode isset up to configure the portable electronic device for playback ofstored image based content. For example, the portable electronic devicecan play back the stored image based content, and the head-mounteddevice can provide the stored image based content in the left and rightimage frames (e.g., left image frame 1110 and right image frame 1112 ofFIG. 11B).

The head-mounted display system may enable the internal viewing mode atany suitable time. In some embodiments, the head-mounted display systemcan automatically begin to operate in the internal viewing mode when theportable electronic device is first coupled to the head-mounted device.For example, the head-mounted display system can include a sensingmechanism for alerting the portable electronic device that it has beeninserted into a head-mounted device. Once alerted, the portableelectronic device can place itself in the internal viewing mode. In someembodiments, the head-mounted display system can enable the internalviewing mode in response to a user instruction (e.g., selecting anappropriate control on the head-mounted device or the portableelectronic device).

Typically, while the head-mounted display system is operating in theinternal viewing mode, the user may have limited awareness of theoutside world. For example, when the user is watching stored image basedcontent on a head-mounted display system, the user may be unaware whensomeone approaches him and asks a question. As a result, thehead-mounted display system may allow the user to interact with hissurroundings by operating in an external viewing mode.

In the external viewing mode, a detecting mechanism (e.g., a camera onthe portable electronic device such as camera 1010 of FIG. 10B or aseparate camera disposed on the outside of the head-mounted device suchas camera 306 of FIG. 3A) can provide for viewing outside of thehead-mounted device when the user is wearing the device. For example,the detecting mechanism can provide external image based content (e.g.,images or video) of the user's surroundings which can be displayed tothe user (e.g., in left image 1110 and right image 1112 of FIG. 11B). Asa result, the user may see his surroundings while wearing thehead-mounted display system.

The head-mounted display system can enable the external viewing modeusing any suitable approach. For example, the head-mounted displaysystem can detect when the user has selected an option to view externalimage based content (e.g., the user may have selected a control, issueda voice command, or made a head movement)

As another example, the head-mounted display system can enable theexternal viewing mode based on an indication received from the user'ssurroundings. For example, sensors on the head-mounted device or theportable electronic device may detect objects or sound around the userand direct the head-mounted display system to enable the externalviewing mode. For instance, in response to a proximity sensor detectingan object in front of the user, the head-mounted display system canautomatically activate the external viewing mode. As another example, inresponse to an ambient sound detector detecting sound around the user,the head-mounted display system can automatically activate the externalviewing mode. To prevent the head-mounted display system fromunnecessarily switching to the external viewing mode whenever sound isdetected, the activation may only occur when sound is detected above acertain threshold (e.g., 10 dB).

In some embodiments, the ambient sound detector can work together with avoice recognition module. For example, the external viewing mode mayactivate if words signaling a possible emergency are detected. Asanother example, the external viewing mode may activate if the voice ofa particular person is detected (e.g., voice of a person on the user'scontact list). Persons skilled in the art will appreciate that either aprocessor on the head-mounted device or a processor on the portableelectronic device can control the activation of the internal viewingmode and the external viewing mode.

In some embodiments, the head-mounted display system can makeadjustments in response to detecting objects or sound. For example,visual cues (e.g., arrows) can be displayed that correspond to thedirection of detected sound. After seeing the cues, the user may selectan option to stop the head-mounted display system from switching to theexternal viewing mode.

In addition, the head-mounted display system may enable the externalviewing mode at any suitable time. For example, the head-mounted displaysystem can automatically switch to the external viewing mode afterwaiting for a period of time. As another example, the head-mounteddisplay system can display a selectable external viewing option inresponse to detecting the appropriate cues. As yet another example, thehead-mounted display system can enable the external viewing mode inresponse to a user instruction received at any other suitable time(e.g., in response to a user selecting an appropriate control on thehead-mounted device or the portable electronic device).

In some embodiments, the head-mounted display system can allow the userto hear sounds from the outside world without activating the externalviewing mode. For example, a microphone on the exterior of thehead-mounted device or the portable electronic device can provide audiothrough the earphones (e.g., earphones 316 of FIG. 3A) of thehead-mounted device. In some embodiments, the head-mounted displaysystem can lower the volume of the image based content that is playingso that the user can hear external sounds. For example, while the useris traveling on an airplane, the ambient sound detector may detect thata flight attendant is asking the user a question. In response todetecting the flight attendant's voice, the head-mounted display systemcan lower the volume of the stored image based content that is playingso that the user can hear the flight attendant's question.Alternatively, in response to detecting sound, the head-mounted displaysystem can mute the stored image based content, and provide closecaptioning of the stored image based content on the display screen. As aresult, the user can see what is being said in the stored image basedcontent even if the sound is not audible. As another alternative, thehead-mounted display system can automatically pause or stop the storedimage based content that is currently playing so that the user can hearexternal sounds. Options can then be provided to either resume playingthe stored image based content or activate the external viewing mode.

In some embodiments, the head-mounted display system can operate in apicture-in-picture (“PIP”) mode, which allows the user to see bothexternal and stored image based content. For example, as shown in FIGS.12A-12C, the head-mounted display system can display a PIP image framein addition to the main image frame(s). The PIP image frame may bespaced apart and utilize a different portion of the screen than the mainimage frame(s) or it may be partially overlaid on top of the main imageframe. Furthermore, the PIP image frame(s) may be smaller than the mainviewing image frames.

Persons skilled in the art will appreciate that the PIP image frame(s)can vary in size (e.g.’ PIP image 1206 of FIG. 12A can vary from 800×600pixels to 200×150 pixels). The user may adjust the size of the PIP imageframe(s) by selecting an option (e.g.’ using controls 318 or remotecontrol 322 of FIG. 3A).

As shown in FIG. 12A, a single PIP image frame 1206 may be placed in thetop right hand corner of the display screen. It may be overlaid over acorner of main image frame 1204 (as shown).

As shown in FIG. 12B, the head-mounted display system can also place aportion of the PIP image frame in each of the main image frames (e.g.,PIP image frames 1208 and PIP image frames 1210). PIP image frames 1208and 1210 can display images in a manner that allows the PIP image framesto be viewed comfortably by the user. For example, PIP image frames 1208and 1210 can be placed in identical locations in left image frame 1212and right image frame 1214. As another example, PIP image frames 1208and 1210 can be offset from each other to adjust for differences in thevisual field of each eye.

As shown in FIG. 12C, the head-mounted display system can display a PIPimage frame 1216 that is separate from main image frames 1218 and 1220.That is, the PIP image frame 1216 is spatially apart and in a differentportion of the display screen.

Because main image frames shown in FIGS. 12A-12C can display storedimage based content and the PIP image frame can display image basedcontent from a camera, the user may continuously see activitiesoccurring in the outside world while watching stored image based contenton the head-mounted display system. It will be understood that imagebased content displayed in the main image frame(s) and the image basedcontent displayed in the PIP image frame(s) can be switched. Thus, mainimage frame(s) can display image based content from the camera and thePIP image frame(s) can display stored image based content playing on thehead-mounted device. The head-mounted display system can provide optionsfor the user to switch between these two PIP modes.

In some embodiments, when the head-mounted display system is operatingin the PIP mode, the user may select an option to cause the head-mounteddisplay system to operate in the external viewing mode. For example, ifthe PIP image frame is not providing a good resolution, the user maywant to enlarge the PIP image frame to full screen. In response to theuser selecting an option to enlarge the PIP image frame, thehead-mounted display system can display the image based content from thedetecting mechanism in the left and right image frames. The head-mounteddisplay system can then remove the PIP image frame.

The PIP mode can be activated (e.g., PIP image frame(s) placed on thedisplay screen) in various ways. For example, the PIP mode can beactivated in response to a user input. Suitable user inputs can include,for example, selecting an option using controls 318 or remote control322, issuing a voice command, or making a head movement. This allows thehead-mounted display system to seamlessly switch from the internalviewing mode or the external viewing mode to the PIP mode. As anotherexample, the PIP mode can be activated when sensors on the head-mounteddisplay system detect sound or objects around the user (e.g., instead ofactivating the external viewing mode discussed above). Thus, while thehead-mounted display system is operating in the PIP mode, the system candisplay what is occurring in the outside world without interrupting theimage based content that is currently playing.

Finally, the PIP mode can be deactivated (e.g., PIP image(s) removed) inresponse to a user selection or if no user selection is received after acertain period of time.

FIG. 13 shows a simplified block diagram of an illustrative head-mounteddisplay system 1300 in accordance with one embodiment of the invention.Head-mounted display system 1300 may include a head-mounted device 1304that connects to a portable electronic device 1302. The system may, forexample, correspond to the systems shown in FIGS. 1-12C.

For the sake of brevity, portable electronic device 1302 will not bedescribed in detail in FIG. 13. It should be appreciated, however, thatportable electronic device 1302 may be embodied in a variety of formsincluding a variety of features, all or some of which can be utilized byhead mounted display system 1300 (e.g., input/output, controls,processing, battery, etc.). In most cases, portable electronic device1302 follows a handheld form factor (e.g., small portable electronicdevice which is light weight, fits in a pocket, etc.). Although notlimited to these, examples include media players, phones (includingsmart phones), PDAs, computers, and the like. Portable electronic device1302 may include screen 1303 for presenting the graphical portion of themedia to the user. Screen 1303 can be utilized as the primary screen ofhead-mounted display system 1300.

Referring to head-mounted device 1304, the device can be arranged to beworn on the head of a user. Head-mounted device 1304 may, for example,correspond to glasses, head gear, goggles and/or the like. Head-mounteddevice 1304 may be configured to receive and carry portable electronicdevice 1302 when they are operatively coupled together. Thus,head-mounted display system 1300 (including portable electronic device1302) can be supported on a user's head. Head-mounted device 1304 and/orportable electronic device 1302 may be configured to be small and lightweight for this reason.

Head-mounted device 1304 can include dock 1306 operative to receiveportable electronic device 1302. In some embodiments, dock 1306 caninclude a connector (e.g., 30 pin, USB, firewire, power, DVI, etc.),which can be plugged into a complementary connector of portableelectronic device 1302. Dock 1306 may include features for helping toalign the connectors during engagement and for physically couplingportable electronic device 1302 to head-mounted device 1304. Forexample, dock 1306 may define a cavity for placement of portableelectronic device 1302. Dock 1306 may also include retaining featuresfor securing portable electronic device 1302 within the cavity.

The connector on dock 1306 can function as the communication interfacebetween portable electronic device 1302 and head-mounted device 1304.Thus, once portable electronic device 1302 is physically coupled tohead-mounted device 1304, head-mounted device 1304 and portableelectronic device can cooperate to form head-mounted display system1300. By way of example, they can share features and functions with oneanother. Their relationship with one another can be master/slave,server/client, peer to peer, and the like depending on the desired needsof the system. In one example, head-mounted device 1304 is thecontrolling device of the system. In another example, portableelectronic device 1302 is the controlling device of the system. In yetanother example, the controlling device depends on the feature/functionbeing used. In other examples, the devices can work together to controlvarious combined aspects of the system (e.g., peer to peerrelationship).

As discussed previously, head-mounted device 1304 can provide additionalfunctionality to portable electronic device 1302. The number ofadditional features provided by head-mounted device 1304 may be widelyvaried. In one example, head-mounted device 1304 can include a varietyof controls (e.g., controls 1308) that control the system and providefunctions to the system as well as access to functions on portableelectronic device 1302. Controls 1308 can be in the form of buttons,switches, dials, and touch or near touch sensing devices (e.g., touchsensors).

In some embodiments, controls 1308 can include a voice recognitionmodule, which can detect and process voice commands that can be used tocontrol the portable electronic device. For example, portable electronicdevice 1302 can use a microphone (e.g., microphone 1016 or microphone1020 of FIG. 10B) to convert sound into electrical signals. The voicerecognition module can then process the electrical signals and identifythe signals as commands for head-mounted display system 1300. Forexample, in response to receiving a voice command to call a contact on auser's contact list, portable electronic device 1302 can automaticallydial the contact's phone number. As another example, in response toreceiving a voice command to stop playing image based content, portableelectronic device 1302 can stop the image based content that iscurrently playing.

In addition, the voice recognition module can be operative to recognizethe voices of multiple people (e.g., family and friends in the user'scontact list). For example, the voice recognition module can be trainedby using voice samples from each person. The training can improve orexpand as head-mounted display system 1300 collects more voice samplesover time. Persons skilled in the art will appreciate that any suitabletechnology can be used to train the voice recognition module, such ashidden Markov models, neural networks, pattern matching algorithms,decision trees, or any combination thereof.

Head-mounted device 1304 may include processor 1310. Processor 1310 canbe connected to any component in the system, for example, via a bus, andcan be configured to perform any suitable function, such as audio andvideo processing. For example, processor 1310 can acoustically enhanceimage based content by configuring speakers 1312 (e.g., haptics) toproduce vibrations and other sound effects corresponding to audio eventsin the image based content. As another example, processor 1310 canconvert (and encode/decode, if necessary) data, analog signals, andother signals (e.g., physical contact inputs, physical movements, analogaudio signals, etc.) into digital data, and vice-versa.

In some embodiments, processor 1310 can receive user inputs fromcontrols 1308 and execute operations in response to the inputs. Forexample, processor 1310 can be configured to receive sound from themicrophone. In response to receiving the sound, processor 1310 can runthe voice recognition module to identify voice commands.

Head-mounted device 1304 can include an optical subassembly 1314configured to help optically adjust and correctly project the imagebased content being displayed by screen 1303 for close up viewing (asdiscussed in FIGS. 6A-6B).

Head-mounted device 1304 may include memory 1318. Memory 1318 can be oneor more storage mediums, including for example, a hard-drive, cache,flash memory, permanent memory such as read only memory (“ROM”),semi-permanent memory such as random access memory (“RAM”), any othersuitable type of storage component, or any combination thereof. In someembodiments, memory 1318 can provide additional storage for image basedcontent that can be played back (e.g., audio, video, and games). Forexample, when a user couples portable electronic device 1302 intohead-mounted device 1304, the user can select to play a movie stored onhead-mounted device 1304. In response to the user selecting to play amovie, the movie can be loaded or streamed to portable electronic device1302, which can play back the movie for the user. In some embodiments,the movie can be copied into memory on portable electronic device 1302.As a result, the user may watch the movie later even after portableelectronic device 1302 has been uncoupled from head-mounted device 1304.In some embodiments, the movie can be removed from portable electronicdevice 1302 as soon as the user selects to stop playing the movie, whichcan free memory on portable electronic device 1302.

Head-mounted device 1304 can include battery 1320, which can chargeand/or power portable electronic device 1302 when portable electronicdevice 1302 is coupled to head-mounted device 1304. As a result, thebattery life of portable electronic device 1302 can be extended.

Head-mounted device 1304 can include cooling system 1322, which caninclude any suitable component for cooling down portable electronicdevice 202. Suitable components can include, for example, fans, pipesfor transferring heat, vents, apertures, holes, any other componentsuitable for distributing and diffusing heat, or any combinationthereof. Cooling system 1322 may also or instead be manufactured frommaterials selected for heat dissipation properties. For example, theframe or housing of head-mounted device 1304 may be configured todistribute heat away from portable electronic device 1302 and/or theuser.

Head-mounted device 1304 can include one or more sensors 1324 to detectvarious signals. Suitable sensors can include, for example, ambientsound detectors, proximity sensors, accelerometers, light detectors,cameras, and temperature sensors. An ambient sound detector can aid theuser with hearing a particular sound. For example, when the user iswatching image based content on head-mounted device 1304, head-mounteddevice 1304 can automatically adjust the volume of the image basedcontent based on the ambient noise detected by the ambient sounddetector. If the volume of the ambient noise is greater than a certainthreshold (e.g., 10 dB), head-mounted device 1304 can increase thevolume of the image based content. Similarly, if the volume of theambient noise is below a certain threshold, head-mounted device 1304 canlower the volume of the image based content. By lowering the volume,head-mounted device 1304 can prevent the sound of the image basedcontent from bothering other people sitting near the user. As anotherexample, if the user selects to listen to outside sounds (e.g., byselecting an option on portable electronic device 1302 or head-mounteddevice 1304), head-mounted device 1304 can lower the volume of the imagebased content. In some embodiments, the ambient sound detector candetect the direction of voices or other sounds that are near the user.

Since the ambient sound detector may detect many sounds that may simplybe random noise, the detector may not transmit the sound to head-mounteddevice 1304 unless one or more conditions are satisfied. For example,the ambient sound detector can determine to transmit the sound if thevolume of the sound is greater than a certain threshold. As anotherexample, the ambient sound detector can work in conjunction with thevoice recognition module to transmit the sound when certain words aredetected (e.g., words signaling a possible emergency or wordscorresponding to the user's name). To identify the detected words, theambient sound detector can attempt to match the words to a storedlibrary of words. In addition, the ambient sound detector can attempt toidentify phrases by matching the words to a stored library of phrases.Finally, as yet another example, the sound may be transmitted if thevoice matches a stored voiceprint. For example, the voice may match astored voiceprint that is associated with a person in the user's contactlist.

A proximity sensor can detect one or more objects that are around theuser (e.g., within a predetermined distance from the user). For example,if the user is wearing head-mounted device 1304 while on an airplane,the proximity sensor can detect when a flight attendant approaches theuser.

Accelerometers on head-mounted device 1304 can detect the user's headmovements. For example, the user may use certain head movements tocontrol head-mounted display system 1300. In response to receiving ahead movement corresponding to a command (e.g., the user tilting hishead to the right for a certain period of time), head-mounted device1304 can execute the associated operation. In some embodiments,head-mounted device 1304 can provide options allowing the user toassociate a particular head movement with a command for controlling anoperation of head-mounted display system 1300 (e.g., a playback commandor a volume command). For example, a head tilt to the right can beassociated with a command to stop playback of the image based content.

In some embodiments, because the user may inadvertently make small headmovements, head-mounted device 1304 can be operative to ignore smallhead movements and only respond to larger head movements. A moredetailed discussion on controlling a head-mounted device using headmovements can be found, for example, in Tang, U.S. patent applicationSer. No. 12/072,763, Attorney Docket No. P5311US1, filed on Feb. 27,2008, entitled “Automatically Adjusting Media Display in a PersonalDisplay System,” which is incorporated by reference in its entirety.

Sensors 1324 can include a camera which can capture image based contentof the outside world. The camera on head-mounted device 1304 can besimilar to the camera on the portable electronic device. In someembodiments, the camera on head-mounted device 1304 may include a biggerlens and provide better resolution than the camera on the portableelectronic device (e.g., because more space is available on head-mounteddevice 1304).

Head-mounted device 1304 can include input/output 1326, which caninclude any suitable component for connecting head-mounted device 1304to other devices. Suitable components can include, for example,audio/video jacks, data connectors, or any additional or alternativeinput/output components.

Head-mounted device 1304 can include communications circuitry 1328 forcommunicating with one or more servers or other devices using anysuitable communications protocol. For example, communications circuitry1328 can support Wi-Fi (e.g., a 802.11 protocol), Ethernet, Bluetooth™(which is a trademark owned by Bluetooth Sig, Inc.), high frequencysystems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems),infrared, TCP/IP (e.g., any of the protocols used in each of the TCP/IPlayers), HTTP, BitTorrent, FTP, RTP, RTSP, SSH, any other communicationsprotocol, or any combination thereof. Communications circuitry 1328 canalso include an antenna for transmitting and receiving electromagneticsignals.

Alternatively or in addition to a connector on dock 1306, thecommunication interface can also be supported by communicationscircuitry 1328, which can wirelessly connect portable electronic device1302 and head-mounted device 1304. Thus, head-mounted device 1304 cancommunicate wirelessly with portable electronic device 1302. By way ofexample, mechanisms such as RF, Bluetooth™, 802.11 and the like can beused.

Communications circuitry 1328 can also use any appropriatecommunications protocol to communicate with a remote server. The remoteserver can be a database that stores various types of image basedcontent. When head-mounted device 1304 is connected to the remoteserver, image based content (e.g., videos and songs) can be downloadedto portable electronic device 1302 or head-mounted device 1304 forplayback. The image based content can be stored on portable electronicdevice 1302, head-mounted device 1304, or any combination thereof. Inaddition, the stored image based content can be removed once playbackhas ended.

It should be appreciated that the head-mounted device can includeelements in addition to the described elements, such as microphones,secondary displays, and indicators. It should also be appreciated thatany combination of the described elements may be provided in thehead-mounted device. It should further be appreciated that, in order toprevent redundancies, these elements may alternatively be found in theportable electronic device and therefore may not be provided in thehead-mounted device. It should also be appreciated that in some casesredundancies may work to enhance the system. For example, both devicesmay include cameras that can work together to gather information, oralternatively can work separately to gather and provide differentfeedback.

FIG. 14 shows a flowchart of an illustrative process for displayingimage based content on a portable electronic device in accordance withone embodiment of the invention. In the discussion below, thehead-mounted display system includes a head-mounted device (e.g.,head-mounted device 1304 of FIG. 13) and a portable electronic device(e.g., portable electronic device 1302 of FIG. 13) that is coupled tothe head-mounted device.

Process 1400 starts at step 1402. At step 1404, the head-mounted displaysystem can detect the connection between the head-mounted device and theportable electronic device. For example, the connection can either bewired (e.g., via a connector on dock 1306 of FIG. 13) or wireless (e.g.,via communications circuitry 1328 of FIG. 13). After a connection hasbeen detected, process 1400 moves to step 1406.

At step 1406, the head-mounted display system can adjust image basedcontent displayed on the portable electronic device for close upviewing. Step 1406 will be described in more detail in connection withFIG. 15. After the image based content has been adjusted, process 1400moves to step 1408.

At step 1408, the head-mounted display system can display the adjustedimage based content to the user. For example, a display screen on thehead-mounted device can project the adjusted image based content to theuser. Process 1400 then ends at step 1410.

FIG. 15 shows a flowchart of an illustrative process for adjusting imagebased content displayed on a portable electronic device in accordancewith one embodiment of the invention. In some embodiments, process 1500can be implemented in step 1406 of process 1400 (FIG. 14). Althoughsteps 1502 to 1510 are shown in a certain order in FIG. 15, the stepsmay be implemented in any order.

Process 1500 starts at step 1502. At step 1504, the head-mounted displaysystem can determine the number of image frames to be displayed. Forexample, if multiple image frames are to be displayed, the head-mounteddisplay system can break a single image frame into multiple image frames(see FIG. 11B). Multiple image frames may, for example, be used toimplement stereo-like viewing. After properly configuring the number ofimage frames to be displayed on the portable electronic device, process1500 moves to step 1506.

At step 1506, the head-mounted display system can adjust the size of theone or more image frames. For example, because the screen of theportable electronic device may be placed fairly close to the user'seyes, the image based content may be adjusted to be less than the sizeof the screen (see FIG. 11A). After adjusting the size of the one ormore image frames, process 1500 moves to step 1508.

At step 1508, the head-mounted display system can adjust the position ofthe one or more image frames. For example, in order to align the imageswith the user's eyes, the image frames may be adjusted either togetheror relative to one another (see FIGS. 11C and 11D). After adjusting theposition of the image frames, process 1500 moves to step 1510.

At step 1510, the head-mounted display system can adjust additionalcharacteristics of the image frames. For example, the head-mounteddisplay system can adjust the resolution of the image based content. Forinstance, the resolution may be increased to compensate for how closethe eyes are to the display screen. After the additional characteristicsof the images have been adjusted, process 1500 ends at step 1512.

FIG. 16 shows a flowchart of an illustrative process for displayingimage based content on a head-mounted device in accordance with oneembodiment of the invention. Process 1600 starts at step 1602.

At step 1604, the head-mounted device can receive one or more imageframes from the portable electronic device. At step 1606, thehead-mounted device can determine whether the portable electronic devicehas split its screen into a left image frame and a right image frame.If, at step 1606, the head-mounted device determines that the portableelectronic device has split its screen into a left image frame and aright image frame, process 1600 moves to step 1608.

If, at step 1606, the head-mounted device instead determines that theportable electronic device has not split its screen into a left imageframe and a right image frame, process 1600 moves to step 1610. At step1610, the head-mounted device can generate left and right image framesfrom a single image frame received from the portable electronic device.For example, an optical subassembly on the head-mounted device (e.g.,optical subassembly 1314 of FIG. 13) can include two optical interfaces.Each optical interface can generate and display an image frame (leftimage frame or right image frame) associated with the received imageframe. After generating the left image frame and the right image frame,process 1600 moves to step 1608.

At step 1608, the head-mounted device can determine whether the left andright image frames are aligned with the user's eyes. For example, thehead-mounted device can detect the interpupillary distance between theuser's eyes and determine if the separation between the left and rightimage frames is appropriate. If, at step 1608, the head-mounted devicedetermines that the left and right image frames are aligned with theuser's eyes, process 1600 moves to step 1612.

If, at step 1608, the head-mounted device instead determines that theleft and right image frames are not aligned with the user's eyes,process 1600 moves to step 1614. At step 1614, the head-mounted devicecan displace and align the left and right image frames with the user'seyes. For example, the head-mounted device can use the opticalsubassembly to align the two image frames. As another example, thehead-mounted device can direct the portable electronic device to affectthe left and right image frames to align with the user's eyes. Afteraligning the image frames, process 1600 moves to step 1612.

At step 1612, the head-mounted device can determine whether the left andright image frames are separated by an overlay. For example, theportable electronic device may have superimposed a black overlay over anarea adjacent to the periphery of the left and right image frames. If,at step 1612, the head-mounted device determines that the left and rightimage frames are separated by an overlay, process 1600 ends at step1616.

If, at step 1612, the head-mounted device instead determines that theleft and right image frames are not separated by an overlay, process1600 moves to step 1618. At step 1618, the head-mounted device caninclude a black stencil defining image frames on the display screen ofthe head-mounted device for the left image frame and the right imageframe. Process 1600 then ends at step 1616.

FIGS. 17A and 17B show flowcharts of an illustrative process fordisplaying external image based content in accordance with oneembodiment of the invention. Process 1700 starts at step 1702.

At step 1704, the head-mounted display system can determine whether auser input has been received from controls to display the user'ssurroundings. For example, the user may have selected controls 318 (FIG.3A) or remote control 322 (FIG. 3A). As another example, the controlscan include a voice recognition module which can be operative toidentify if the user has issued a voice command. If, at step 1704, thehead-mounted display system determines that a user input has beenreceived from the controls, process 1700 moves to step 1706.

If, at step 1704, the head-mounted display system instead determinesthat a user input has not been received from the controls, process 1700moves to step 1708. At step 1708, the head-mounted display system candetermine whether a user input has been detected from sensors. Forexample, accelerometers on the head-mounted display system can detect ifthe user has made any head movements. Based on the detection of aparticular head movement, the head-mounted display system can determineif the head movement is an indication that the user would like to viewimage based content from the outside world. If, at step 1708, thehead-mounted display system determines that a user input has beenreceived from the sensors, process 1700 moves to step 1706.

If, at step 1708, the head-mounted display system instead determinesthat a user input has not been received from the sensors, process 1700moves to step 1710.

At step 1710, the head-mounted display system can determine whether anexterior condition has been satisfied for displaying the user'ssurroundings. For example, proximity sensors on the head-mounted displaysystem may detect one or more objects around the user (e.g., detect theobjects within a predetermined distance from the user). As anotherexample, an ambient sound detector may detect sound that is above acertain threshold. As yet another example, the ambient sound detectormay work together with the voice recognition module to detect certainwords or the voice of a particular person. If, at step 1710, thehead-mounted display system determines that an exterior condition hasnot been satisfied to display image based content of the user'ssurroundings, process 1700 ends at step 1712.

If, at step 1710, the head-mounted display system instead determinesthat an exterior condition has been satisfied to display image basedcontent of the user's surroundings, process 1700 moves to step 1706.

At step 1706, a PIP image frame overlaid on at least one of displayedleft and right image frames (e.g., PIP mode) can be displayed. Forexample, the left and right image frames can correspond to stored imagebased content playing on the portable electronic device, and the PIPimage frame (e.g., PIP image frame 1206 or PIP image frames 1208 and1210 of FIGS. 12A and 12B) can correspond to external image basedcontent received from the user's surroundings. The external image basedcontent can, for example, be captured by a detecting mechanism on theportable electronic device or the head-mounted display system.

In some embodiments, if one or more exterior conditions has beensatisfied (e.g., sound has been detected), the head-mounted displaysystem can perform other functions in addition to displaying the PIPimage frame. For example, ambient sound sensors on the head-mounteddisplay system can detect that the external sounds are coming from acertain direction. In response to detecting the direction, thehead-mounted display system can display arrows corresponding to thedirection of the sound. As another example, the volume of the storedimage based content can be lowered so that the user can hear externalsounds, and a microphone on the exterior of the head-mounted displaysystem can provide audio through the earphones of the head-mounteddisplay system. To ensure that the user does not miss any of the imagebased content being played back, close captioning corresponding to thestored image based content can be displayed. Alternatively, the storedimage based content that is currently playing can be stopped or pausedso that the user can hear the external sounds. Process 1700 then movesto step 1714.

At step 1714, the head-mounted display system can determine whether auser instruction to switch to the image based content in the PIP imageframe was received. For example, the user may select a control, issue avoice command, or make a head movement. If, at step 1714, thehead-mounted display system determines that a user instruction to switchto the image based content in the PIP image frame has not been received,process 1700 moves to step 1716.

If, at step 1714, the head-mounted display system instead determinesthat a user instruction to switch to the image based content in the PIPimage frame has been received, process 1700 moves to step 1718. At step1718, the head-mounted display system can switch to the image basedcontent in the PIP image frame (e.g., external viewing mode) and removethe PIP image frame. In some embodiments, the head-mounted displaysystem can switch to the image based content in the PIP image frameautomatically without waiting for a user selection. For example, afterdisplaying arrows corresponding to the direction of external sounds, thehead-mounted display system can immediately switch to the externalviewing mode. After switching the image based content, process 1700moves back to step 1704. At step 1704, the head-mounted display systemcan determine whether to display the PIP image frame again based on oneor more inputs.

At step 1716, the head-mounted display system can determine whether auser instruction to remove the PIP image frame has been received. If, atstep 1716, the head-mounted display system determines that a userinstruction to remove the PIP image frame has been received, process1700 moves to step 1720.

If, at step 1716, the head-mounted display system instead determinesthat a user instruction to remove the PIP image frame has not beenreceived, process 1700 moves to step 1722.

At step 1722, the head-mounted display system can determine if thedisplay of the PIP image frame has timed-out. For example, thehead-mounted display system can set a time period to wait for a userinstruction to remove the PIP image frame. If, at step 1722, thehead-mounted display system determines that the display of the PIP imageframe has not timed-out, process 1700 moves back to step 1716.

If, at step 1722, the head-mounted display system instead determinesthat the display of the PIP image frame has timed-out, process 1700moves to step 1720. At step 1720, the PIP image frame can be removed,and the head-mounted display system can return to the internal viewingmode. After removing the PIP image frame, process 1700 moves back tostep 1704. At step 1704, the head-mounted display system can determinewhether to display the PIP image frame again based on one or moreinputs. For example, the user may have been preoccupied with watchingthe stored image based content, and, as a result, was not able to switchto the image based content in the PIP image frame before the image framewas removed. Now, however, the user may want to view the external imagebased content. As a result, the user may go back and forth betweenhaving the PIP image frame displayed and/or removed.

FIG. 18 illustrates a process 1800 for displaying images in ahead-mounted display apparatus to a user. The method may for example beaccomplished via communications between a processor, memory and adisplay screen of the head mounted display apparatus. In one embodiment,the head mounted display apparatus is a single device that includes itsown display. In another embodiment, the head mounted display apparatusmay include multiple discrete devices that cooperate together to form asingle unit. For example, the head mounted display apparatus may includea portable electronic device such as media player or phone and ahead-mounted device that carries the portable electronic device.

Process 1800 starts at step 1802 where stored content such as mediacontent is displayed. For example, stored content can include movies, TVshows, music videos, or software such as applications or operatingsystems.

After displaying stored image based content, process 1800 moves to step1804. At step 1804, external content from the user's surroundings can becaptured. External content may, for example, include visual or soundcontent. The external content may be captured using a variety of devicesincluding a camera and/or a microphone.

After capturing external content, process 1800 moves to step 1806. Atstep 1806, the external content is displayed. For example, visual datacaptured by a camera may be displayed. In addition, sound data may alsobe displayed. For example, speech may be converted to displayed text.Alternatively or additionally, visual indicators associated with thesound data may be displayed.

In one embodiment, the external content and the stored content aresimultaneously displayed. In one example, the stored content isdisplayed in a primary image frame while the external content isdisplayed in a secondary image frame. The secondary image frame may forexample be a picture in picture (“PIP”) image frame. In another example,the external content is displayed in a primary image frame while thestored content is displayed in a secondary image frame. In anotherembodiment, the external content interrupts the display of the storedcontent. For example, the external content may replace the storedcontent (e.g., the playing of the stored content may be paused while theexternal content is being captured and displayed).

Various enhancements may be applied to the present invention to furtherimprove the head-mounted device. For example, the invention may providemethods and apparatus for providing a wider field of view and creating amore natural viewing situation for a user of a head-mounted display.This may result in improved comfort and usability for head mounteddisplays as described in U.S. Published Patent Application No.2008/0088529, entitled “ENHANCED IMAGE DISPLAY IN HEAD-MOUNTEDDISPLAYS”, published Apr. 17, 2008, which is incorporated by referenceherein in its entirety.

The invention may also provide methods and apparatus for treating theperipheral area of a user's field of view in a head-mounted display.This helps to create improved comfort and usability for head-mounteddisplays as described in U.S. Published Patent Application No.2008/0088936, entitled “PERIPHERAL TREATMENT FOR HEAD-MOUNTED DISPLAYS”,published on Apr. 17, 2008, which is incorporated by reference herein inits entirety.

FIG. 19 illustrates a process 1900 for displaying images in ahead-mounted display apparatus to a user. The process 1900 includesblock 1910 where stored media is displayed. The process also includesblock 1920 where the user's surroundings are detected and at least oneexterior condition is satisfied. The process also includes block 1930where external media from the user's surroundings is captured inresponse to detecting the user's surroundings. The process furtherincludes block 1940 where the captured external media is displayed.

In some embodiments, wherein detecting in block 1920 further comprisesat least one of: detecting at least one object within a predetermineddistance from the user; detecting at least one sound that exceeds aparticular threshold; detecting at least voice matching a storedvoiceprint (e.g., voiceprint associated with a person in user's contactlist); detecting at least one word matching a particular word in astored library of words (e.g., user's name); and detecting at least onephrase matching a particular phrase in a stored library of phrases.

In some embodiments, block 1910 further includes displaying left andright images corresponding to the stored media.

In some embodiments, displaying the captured external media of block1940 comprises replacing the stored media with the captured externalmedia. Replacing the stored media may include displaying left and rightimages corresponding to the captured external media.

In some embodiments, displaying the captured external media furtherincludes displaying the captured external media in a PIP window.

FIG. 20 illustrates a process 2000 performed on a portable electronicdevice with a screen. The process includes in a first display mode 2010,presenting image based content according to a first format. The firstformat has a first resolution and configures the screen for normalviewing. The process also includes in a second display mode 2020,presenting image based content according to a second format. Wherein thesecond format has a second resolution and configures a portion of thescreen for close up viewing. The process additionally includes block2030 which comprises detecting that the portable electronic device isbeing used for normal viewing and switching to the first display mode.The process further includes block 2040 which comprises detecting thatthe portable electronic device is being used for close up viewing, andswitching to the second display mode.

FIG. 21 is a method for displaying one or more image frames on a screenof a portable electronic device when the portable electronic device isconnected to a head-mounted device. The method 2100 includes receivingan image frame (block 2110). The method also includes dividing the imageframe into a left image frame and a right image frame (block 2120). Themethod further includes formatting the left and right image frames forclose up viewing (block 2130). The method additionally includesdirecting the left image frame to the left portion of the screen anddirecting the right image frame to the right portion of the screen(block 2140).

In some embodiments dividing the image frame (block 2120) includesduplicating the image frame to form the left and right image frames.

In some embodiments, the sizes of the left and right image frames aremade smaller than the size of the screen.

The invention may further provide other features of head-mounted devicesas described in co-pending U.S. patent application Ser. No. 12/114,499,entitled “HEAD MOUNTED DISPLAY”, filed on May 2, 2007, which isincorporated by reference herein in its entirety.

The invention is preferably implemented by hardware, software or acombination of hardware and software. The software can be embodied ascomputer readable code on a computer readable medium. The computerreadable medium is any data storage device that can store data which canthereafter be read by a computer system. Examples of the computerreadable medium include read-only memory, random-access memory, CD-ROMs,DVDs, magnetic tape, optical data storage devices, and carrier waves.The computer readable medium can also be distributed overnetwork-coupled computer systems so that the computer readable code isstored and executed in a distributed fashion.

Thus it is seen that systems and methods are provided for allowing usersto couple a portable electronic device in the head-mounted device. It isalso seen that systems and methods are provided for allowing users tosee the outside world while wearing a head-mounted device. Personsskilled in the art will appreciate that the invention can be practicedby other than the described embodiments, which are presented forpurposes of illustration and not of limitation, and the presentinvention is limited only by the claims which follow.

What is claimed is:
 1. A head-mounted apparatus comprising: a supportstructure; a display supported by the support structure; a cameraconfigured to capture an image; lenses adjacent to the display throughwhich the display is viewable; and a processor configured to operate ina first mode in which content is displayed on the display withoutdisplaying the captured image and in a second mode in which the capturedimage is displayed on the display.
 2. The head-mounted apparatus definedin claim 1, wherein the captured image is overlaid over a portion of thecontent in the second mode.
 3. The head-mounted apparatus defined inclaim 1, wherein the content is overlaid over the captured image in thesecond mode.
 4. The head-mounted apparatus defined in claim 3, whereinthe content comprises game content.
 5. The head-mounted apparatusdefined in claim 1, wherein only the captured image is displayed in thesecond mode.
 6. The head-mounted apparatus defined in claim 1, furthercomprising: a sensor, wherein the processor is configured to operate inthe second mode in response to detection of a stimulus by the sensor. 7.A head-mounted apparatus comprising: a support structure; a displaysupported by the support structure; a camera configured to capture animage; lenses adjacent to the display through which the display isviewable; and a processor configured to: operate in a first mode inwhich content is displayed on the display without displaying thecaptured image; and operate in a second mode in which both the contentand the captured image are simultaneously displayed on the display. 8.The head-mounted apparatus defined in claim 7, wherein both the contentand the captured image are simultaneously displayed on the display inthe second mode by overlaying the captured image over a portion of thecontent.
 9. The head-mounted apparatus defined in claim 7, wherein boththe content and the captured image are simultaneously displayed on thedisplay in the second mode by overlaying the content over the capturedimage.
 10. The head-mounted apparatus defined in claim 9, wherein thecontent comprises game content.
 11. The head-mounted apparatus definedin claim 7, wherein the processor is further configured to: operate in athird mode in which the captured image is displayed on the displaywithout displaying the content.
 12. The head-mounted apparatus definedin claim 7, wherein the content comprises text.
 13. The head-mountedapparatus defined in claim 7, wherein the content comprises a graphicaluser interface associated with a software application.
 14. Thehead-mounted apparatus defined in claim 7, wherein the content comprisesimage based content associated with a software application.
 15. Thehead-mounted apparatus defined in claim 7, wherein the content comprisesgame content.
 16. The head-mounted apparatus defined in claim 7, furthercomprising: a proximity sensor, wherein the processor is configured toswitch from the first mode to the second mode in response to detectionof an object by the proximity sensor.
 17. A head-mounted apparatuscomprising: a support structure; a display supported by the supportstructure; a camera configured to capture an image; lenses adjacent tothe display through which the display is viewable; and a processorconfigured to operate in an internal viewing mode in which the capturedimage is not displayed, wherein the processor is configured to switchfrom the internal viewing mode to an external viewing mode in which thecaptured image is displayed on the display in response to a stimulus.18. The head-mounted apparatus defined in claim 17, further comprising:a proximity sensor that is configured to detect the stimulus.
 19. Thehead-mounted apparatus defined in claim 17, further comprising: aspeaker that is configured to provide audio at a volume that is loweredin response to the stimulus.
 20. The head-mounted apparatus defined inclaim 17, wherein content is displayed on the display in the internalviewing mode, the head-mounted apparatus further comprising: amicrophone configured to capture audio; and a speaker that is configuredto provide audio that is associated with the displayed content in theinternal viewing mode and to provide the captured audio from themicrophone in the external viewing mode.